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ABSTRACT 

A project tested the feasibility of describing 
learning outcomes in adult/continuing education in ways that could be 
more clearly understood by three major groups of people — stu^'ents, 
teachers, and employers of graduates—and how these could be assessed 
and accredited. Academic staff explored four different approaches to 
describing learning outcomes based on . i ect i ves , subject knowledge, 
discipline, and competence. Student views were collected through 
structured workshops. Students described personal, social, academic, 
and vocational outcomes but were unable to write their own learning 
outcomes. Employer views were collected through consultation, 
interviews, and discussion. Employers judged employability of 
graduates through a small number of generic skills. A workshop on 
assessment showed little evidence of development of an assessment 
strategy for a course or program as well as the use of conservative 
approaches. Other work on accreditation raised three major issues: 
level, credit value, and coherence. Work on accreditation undertaken 
by three project subject groups resulted in three models basod on 
general, subject specific, and core learning outcomes. A four-stage 
model was proposed: outcome des cription, assessment, curriculum 
model, and credit rating. (Notes include information on taxonomies, 
learning styles, and knowledge; approaches to assessment; and credit 
accumulation and transfer systems. Appendixes include project 
strategy and methodology, participant lists, and a 31-item 
bibliography.) (YLB) 
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Preface 



Tlie first degree is the benchmark around which the British education system 
has traditionally been built. Higher Education is defined as education leading to 
it, or to what follows from it; the A level qualification which dominates the 
final years of secondary education was designed to control entry to it; the O 
level qualification in turn was designed to control entry to A level, and so the 
influence cascaded down to the earliest years of schooling. At the same time, 
while the first degree is not regarded as a directly vocatic*^'' qualification, it 
exerts considerable influence on the labour market. Employers recruitment be- 
haviour clearly reflects a notion that * graduates' are, in some fundamental 
sense, different from non-graduates, and that for most purposes the fact that 
someone is a graduate is more important than the subject of the degree. 

However, despite this pivotal role, the notion of what a degree represents has 
never been very clear. While one might wish to argue that all graduates share a 
broadly common range of qualities - analytical, critical, communicative, etc. it 
is not clear how far these are common across disciplines or institutions, nor is 
it always clear how the processes of teaching and assessment in higher educa- 
tion actually develop and measure them. In the absence of such evidence it is 
equally easy to argue that the first degree exists primarily to sort young people 
into a rank order for social and economic purposes, using a set of criteria 
which at best lack a coherent theoretical underpinning, and at worst, are largely 
arbitrary. 

Pragmatists sometimes argue that the absence of a theoretical base is irrelevant, 
since there is a well established professional consensus about what a graduate 
is. Such an approach rightly recognises that the notion of a first degree does 
represent something real» albeit not precisely described. However, if one ac- 
cepts the notion that first degrees in their present form represent something 
which individuals and the nation need, pressures to increase qualification le- 
vels, reduce costs and at the same time maintain quality put such a professional 
consensus imder considerable strain. Unless it is possible to define more clearly 
and publicly what a first degree represents, one of these forces will inevitably 
have to give way. 

Of the three, quality is the most vulnerable, since expenditure levels and par- 
ticipation rates can both be simply described. On the other hand, a notion of 
quality which is understood only within a closed professional circle is unlikely 
to exert much force against clear and simple numerical data. The long term 
social and economic effect will be the same as a reduction in participation, but 
it will be more damaging and insidious, beacuse it will take much longer to be 
visible. 

We need, therefore, to find better defmitions of what a first degree represents. 
We also need to do this rapidly, since structural changes are already eroding 
the benchmarks on which the pragmatic definitions rest. In practice, the key 
benchmark is the 'traditional' young graduate. The qualities which such people 
have in common are not that they have demonstrated a common set of skills or 
knowledge, since this evidently varies very widely between subjects, institu- 
tions and over time. Nor can it easily be argued that they share a common set 



of underpinning intellectual capabilities, since there is little evidence that such 
things are consciously developed or assessed when decisions are made on 
whether or not to award degrees. 

What is, or used to be, common to all graduates lies not in what they can do, 
but in the fact that they have participated in a common set of processes. They 
are all of broadly the same age, and have survived a lengthy selection process, 
involving formal and informal selection at 14, 16 and 18. They have spent 
three years in *full time* study with a large number of people from similar 
social and economic background, away from home during a critical period of 
their social and emotional development. During that time they have been freed 
from major economic insecurity, provided with teaching at a relatively low 
staff student ratio, and with relatively free access to extensive libraries and 
other learning resources. The notion that a first degree represents something 
distinctive, which differentiates graduates from others would appear (at least) 
to stem as much from these common experiences as from a demonstration of 
particular personal or intellectual qualities, although the former may be sur- 
mised to derive from the latter. 

.Almost all the elements of this process are now being eroded. Increasingly, 
undergraduates are. older with more stable domestic circumstances and respon- 
sibilities, and entering through non-traditional routes of various kinds. They are 
more often studying part-time, living at home, under serious financial stress, 
experiencing high staff student ratios and Limited access to learning resources. 
If the defining characteristics of a graduate arose implicitly from participation 
in the former process, and are not explicitly described, how can we be sure that 
those who participate in an entirely different process are in any sense 'grad- 
uates' at all? The quality challenge is clear; unless we can define what we 
mean by a graduate we can neither prove or disprove arguments about changes 
in quality in response to change in resourcing or circumstances. If the notion of 
higher education is worth defending, it is worth defining. 

Stephen McNair 
Head of Unit 
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1 Project Objectives 

The project described in this report was based on iwo simple premises: 

• that learning is the central purpose of higher education 

• that the measurement of leairdng might best be achieved through the 
description of outcomes (what a learner knows, or can do as a result of 
learning) rather than the more traditional description of learning input 
(syllabus or course content). 

The purpose of the project was to examine the feasibility of describing a de- 
gree in terms of its outcomes - what a graduate can do, know and understand. 

It began fix)m the premise that a degree is currently described in tcnns of the 
process - three years full time study - rather than achievement, and that, as a 
result, notions of quality arc based on the process, rather than its outcome. This 
approach may be increasingly difficult to sustain as many of the elements of 
tlie traditional process are removed or reshaped in the move from an elite to a 
muss higher education system. Unless the outcomes of the traditional model are 
adequately described, it will be impossible to property assess the impact on 
quality of the mmy structural changes taking place. 

The project proposed that a stronger focus on outcomes would enable quality 
assurance processes to be more transparent and rigorous. It was believed that 
(his would help improve the quality of the dialogue between higher education, 
its learners, funders and those who employ its graduates. 

It was also believed Uiat the description of outcomes alone would net solve the 
problem, and that it was essential that those outcomes could be assessed, and 
recognised in systems of accreditation. Credit would thus be more directly 
linked to explicit achievement. A further benefit of the approach would be in 
the increased ability to recognise and accredit prior learning, and learning 
undertaken outside formal classroom settings. 

The project's aim was exploratory and developmental. It sought to find ways of 
recognising the full range and diversity of achievement which existing degree 
programmes represent, not to narrow or constrain them. It therefore set out to 
investigate a range of approaches with practising academics, rather than to im- 
pose a pattern or methodology. 



2 Key Conclusions 



The project work points to the following key conclusions: 

• it is possible to describe the outcomes of higher education more 
explicitly, although they cannot be expressed in simple *can do' 
statements, and, in a complex and changing environment, such definitions 
will never be complete or fixed; 

• descriptions of learning outcomes in higher education cannot be 
expressed as a single set of * national standards' of Ihc kind developed for 
national vocational qualifications, since higher education exists to meet 
the needs of a variety of client groups and a range of social, economic, 
scientific and actual needs, and properly embodies a range of different 
cultures and value systems; 

• it is necessary to develop processes within each institution to link 
outcome definitions with quality assurance, since the authority to define 
the purposes of degree programmes rests with the chartered institution, 
rather than with any national agency; 

• the principal benefit in an outcomes led approach lies, liiercfore, in 
providing a focus for staff, students aiid employei^ to examine more 
clearly what they are seeking to achieve, and enabling them to contribute 
actively to tlie development of a common understanding of the nature and 
purposes of higher education and of specific programmes and awards; 

• the process of defining and accrediting outcomes can best be seen in 
terms of a four stage cycle: 

preliminary definition by academic staff, 
design of an assessment strategy, 

production of 2 * curriculum model' which weights the various 
types of oulf nc - core, subject specific, general and other, 

allocafion of credi: value lO groups of outcomes; 

• a range of materials and approaches exist to help in the definition of 
outcomes. These include models based on objectives, subject knowledge, 
the discipline and competence. No single one has been demonstrated to 
be ideal and there may be advantages in using more than one; 

• if quality is to be assured it is import.ant that the assessment and 
curriculum modelling stages of Uie cycle are carried out before the 
allocation of credit value; 

• the definition of outcomes should not be separated from an investigation 
of assessment since outcome definitions which are not assessable are not 
of any practical value; 

• an outcome led approach requires staff to develop and use methods of 
assessment which measure achievements directly, but current assessment 
practice tends to neglect these questions of validity in favour of 
reliability, and many academic staff lack experience of appropriate 
approaches to assessment; 
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employers arc, in general, more concerned with the 'core* leamdng 
outcomes of higher education than with general or subject specific ones; 

students can play a particularly valuable role in the development of 
approaches to assessment, and this dialogue can itself help to increase 
their motivation and advisement; 

the definition of learning outcomes is considerably helped by the use of 
cross-institutional subject groups, which provide a broader contrast of 
perspectives than is possible in a single institution. Opportunities for this 
kind of staff development appear to be rare. 



3 What was achieved? 

The project has helped to 'flesh out' ways of describing learning outcomes in 
higher education; to clarify who can and should be involved in their develop- 
ment, and in what ways. It achieved two tilings: 

♦ it tested and refined a collaborative model for the development of 
learning outcome*, statements, working by an iterative and collaborative 
process from within the academic culture, rather than by imposing 
managerial models from outside. It used a simple experiential approach to 
help subject specialists to capture the essence of what it means to be "a 
graduate" in a given subject, in the belief that the solution lay in helping 
practising academic staff to make their tacit knowledge and understanding, 
explicit and public. 

• it produced a quality development model which imks the definition and 
assessment of outcomes with an approach to coherence and a credit 
framework. This has important implications for the quality assurance of 
all learning in HE but especially for learning which is not part of a course 
(like some work bavSed learning) and for the accreditation of prior leaming. 



4 Methodology 

The project was undertaken by the Unit for the Development of Adult Conti- 
nuing Education, in collaboration with nine institutions of higher education in 
England and Wales, and funded by the Department of Emplojonent. 

The Project Director and Project Officer worked with academic staff from the 
parlicipadng institudons in cross institudonal subject groups, to define the out- 
comes of exsisting degree programmes, and examine approaches to assessing 
and accrediting those outcomes. Five subject areas were con^dered: Design, 
English, Engineering, Environmental Science and Social Science, 

Ninety three institutions were represented at a consultative conference and the 
results of the preliminary work by the subject groups were published as What 
Can Graduates Do? This was circulated to all higher education insdtudons for 
comment, and 74 responded. An edited version of tlie publication was circu- 
lated for comment to a range of employers, and 138 did so. A series of work- 
shops were held with undergraduates to explore student views, involving some 
700 students in all participating institutions. 
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5 Development Issues 



The project has identified some key issues requiring further attention. These are 
Assessment 

The project has demonstrated the fundamental connection between the descrip- 
tion and assessment of learning outcomes, and the importance of this in main- 
taining quality. The evidence is that many academic staff are not well equipped 
to assess some of the outcomes generally regarded as central to higher educa- 
tion. Development work is required on techniques for assessing some of the 
more complex and difficult of the outcomes of higher education, and staff de- 
velopment on assessment for academic staff generally is also a high priority. 

Level 

As the boundaries between higher education and other forms of education and 
training become more permeable, it becomes more critical to understand how 
level is being defined, and seek a consensus on this. Further work is needed on 
the notion of level within CAT systems. The project suggests that there are at 
least two quite distinct notions of level in use in higher education (based on 
^milestones/prerequisites' and uu * growth/autonomy') and that much of the as- 
sessment of achievement is designed to reflect this, but without any explicit 
agreement about intentions, definitions or techniques. 

Core 

Further work is required to seek clarification of what are the core qualities 
which characterise a "graduate". There is a wide consensus that there is such a 
set but, although a variety of models exist, there is no agreement about what 
these qualities are, nor how they are to be recognised. Both employers and 
academic staff felt that it was currently possible for people to graduate without 
some of the key capabilities which they expect of a graduate. 

Credit 

The relationships between the various credit accumulation and transfer systems 
are complex. Work is needed on the notion of credit, and how it can be defined 
to ease transition between higher, further, adult and work based learning, TTiis 
is particulariy important in relation to the connections between National Voca- 
tional Qualifications and degrees. 

Collaboration 

The relationships between education and training institutions are becoming 
more complex, and learners increasingly seek to move across sectoral and in- 
stitutional boundaries. The definition of learning outcomes may help in articu- 
lating these relationships (and cross sectoral groups may be a valuable way of 
defining outcomes). Work is needed to establish the part which outcome defini- 
tion can play in encouraging collaboration between institutions and agencies, 
and the part which existing collaborative structures, consortia. Open College 
Networks etc. can play in defining outcomes. 



tRLC 



IV 



Section 1 



Aims and Strategy 



Aims and Strategy 



Chapter 1 



Introduction 



1.1 Why Learning Outcomes? 

The project described in this report was based on two simple premises: 

• that learning is the central purpose of higher education, 

• that the measurement of learning might best be achieved through the 
description of outcomes (what a learner knows and can do as a result of 
leaming) rather than the more traditional description of learning input, 
(syllabus or course content). 

The concept of leaming outcomes is not original: it has been used extensively 
in many areas of education and has become a major feature of both the Na- 
tional Cunriculum and of National Vocational Qualifications (NVQs) but its 
relevance to higher education had not been explored. 

The project set out to test how far it is possible to describe the outcomes of 
leaming in higher education in ways which could be more cleariy understood 
by three major groups of people: students, teachers, and employers of grad- 
uates, and how these could be assessed and accredited. 

1.2 Project Rationale 

The rationale and impetus for this project arose from several observations and 
findings, 

• That the notion of *the graduate* is ill defined, and, while there is 
widespread agreement that Lhere is a set of qualities which characterise a 
^graduate*, end ^hich involve some kind of "balance of knowledge, 
skills, creative thought and motivation", it has been claimed that some 
students could meet the requirements of an undergraduate course without 
being required to demonstrate that they possessed such qualities. Previous 
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UDACE work investigated the applications of a profile of ^capabilities' 
for selecting individuals without formal qualifications for admission to 
education and training programmes. ITie capabilities include, for example, 
critical thinking, planning skills, perseverance , initiative and others. 
During the final stages of the work the Director of one participating 
Polytechnic commented that he "wished that he could demonstrate that 
the students leaving the institution could do all these things, never mind 
those entering". 

• Graduate employers, and bodies representing their concerns, suggest that 
many graduates do not possess the intellectual and personal skills 
required for employment. The majoriiy of graduate employers do not 
recruit graduates for their subject expertise, but are rather seeking a 
generic se* of intellectual, practical and personal skills which would 
contribute to the development of their business concerns. This suggested 
that the employer notion of a good graduate was in some way different 
from the ^academic' viev/. ("Higher education is primarily an intellectual 
preparation, it is not a vocational preparation".) 

• The challenge to quality arising from changes in die structure and 
processes of higher education itself. With continuing expansion, and 
growth in numbers of alternatively qualified students in part time, 
modular, credit accumulation and otiier schemes, a graduate will no 
longer bo so easily defined as someone who has followed a three year 
course of study in a single subject. Equally, the traditional approaches to 
teaching in HE, particularly the tutorial system, seem unlikely to survive 
the current plans for expansion. The absence of clear descriptions of what 
the notion of a graduate is makes it difficult to establish how such 
changes in structures are actoially affectiing quality. 

• The need, in response to changing staff:student ratios, for students to take 
greater responsibility for their own learning, enabling academic staff to 
concentrate more on facilitating learning, and on assessment. This 
requires both academic staff and students to be clearer about what it is 
that students are expected to achieve, and they must be helped to find 
ways of learning and achieving the outcomes which are most suitable for 
them. 

• The impact of changing financing structures, which increase die 
investment which individual students make in their own education, are 
likely to make tiiem behave more as ^consumers' than ^beneficiaries' of 
the system. In this context, it will become increasingly important to be 
clear to the student what precisely is being offered by higher education in 
terms of personal outcomes. 
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1.3 Advantages 

A learning outcomes approach proposes a shift of attention from structures and 
methods of course delivery, towards what is actually learned. This has potential 
advantages for higher education (and for all forms of education and training) in 
terms of increased flexibility, quality and learner motivation. 

Accessibility 

Describing the outcome rather than the process (three years full time study in 
HE) or input, (content, syllabus), makes it easier to consider alternative ways 
of achieving an outcome, recognising that people learn in different ways, 
places and times, and at different paces, and that these ways are all legitimate 
if tiiey achieve tiie same outcome. It might therefore help to facilitate processes 
like credit accumulation and credit transfer, and tiie accreditation of work based 
learning and other learning which takes place outside the ^classroom*. 

Quality 

While quality is assessed only in relation to the process of higher education, 
any move towards greater flexibility is liable to endanger quality, and may do 
so undetected. The clearer specification of outcomes might make it easier to 
ensure that quality is protected, by ensuring that comparable outcomes are 
achieved, whatever changes happen to structures and processes. 

Motivation 

Describing higher education awards in terms of learning outcomes might help 
students to see degrees as a set of targets to achieve, raUier than as a set period 
of time in which to study. More public statements of what was to be achieved 
would provide a better basis for students to choose programmes and modules, 
and enable tiieni to concentrate on the demonstration of achievement ratiier 
than attendance on a course. Learning outcome statements could also relate 
more readily to the parts of a degree, enabling learners, and especially adults 
attending in new modes, to achieve each outcome, or sensible group of out- 
comes at his or her own pace and convenience. 

1.4 What are Learning Outcomes? 

The work of the project has helped to elaborate the initial premises and to 
establish more clearly what learning outcomes are, and are not; what they 
should include; and how they should be written. 

1.4.1 Guidelines and 'Standards* 

The project set out to describe the learning outcomes of a range of different 
degree courses. It did not set out to write a set of * national standards' which 
would apply to all courses, ratiier it was intended to be an opportunity to inves- 
tigate ways of describing and assessing learning outcomes to produce gui- 
delines which could be used by others. Learning outcomes in higher education 
are not * standards', in tiie sense in which the word is used in National Voca- 
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tional Qualifications but they can be documented in the same way as those 
standards. Doing so can provide a clear and generally comprehensible descrip- 
tion of what students are expected to know and be able to do in order to be 
awarded a degree. 

Higher education embraces a wide range of values and cultures, even within a 
single institution or discipline. The project found clear differences in the learn- 
ing outcomes expected and achieved by students following courses with the 
same name, reflecting the different objectives and teaching strategies in differ- 
ent HE institutions. The learning outcome statements developed by the project 
were intended as examples which could be incluu^ed, ignored or modified by 
others as appropriate. 

1«4«2 Outcomes and Objectives 

It seemed helpful at the outset of the project to make a distinction between 
learning outcome statements and course objectives. The distinction is in some 
ways an artificial one, as some respondents pointed out. However, it did serve 
to make it clear that learning outcomes of degree courses can be achieved in a 
variety of settings other than a formal HE institution. For the purpose of the 
project it also helped to avoid a debate on course structures. The project work- 
ing groups were drawn .^rom institutions with ver^ different course structures, 
ethos and organisation, as well as different approaches to teaching and learn- 
ing. A discussion of course objectives seemed likely to lead to a series of 
debates about course structure and delivery, which, though important, were not 
the main purpose of the work. 

The wisdom of the decision was confirmed by several responses to the project 
consultative document which pointed out that, while course aims should de- 
scribe ''what you hope the course will achieve, a statement of whxit learners 
should be able to do (or do better) as a result of having worked through the 
course'' (Rowntree 1986), existing practice in writing aims and objectives for 
course documents falls some way short of the theory. In general the documen- 
tation of courses, where it existed, did not provide an adequate description of 
what a graduate was expected to know and to be able to do. Many documents 
were essentially a list of knowledge to be covered. A substantial majority of 
those responding to the consultative document suggested that learning outcome 
statements would be more useful to them, particulariy in the accreditation of 
prior, and workbased, learning. 

Outcomes and Assessment 

The assessment of learning outcomes became a central issue in the project, for 
two reasons. The first was that learning outcome statements would be meaning- 
less unless they could be linked to credits, and that this could only be done by 
way of assessment. They had, therefore to be framed in terms which made 
them capable of assessment. Credits had to be awarded for the successful 
achievement of outcomes, not for the completion of a process. One respondent 
to the project consultative document commented: 

"Although the reasons for looking at learning outcomes 
include curriculum development, the point could be made that 
clearly defined learning outcornes enable a review of teaching 
methods to ensure that ad'^quate opportunities are provided 
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for students to practice and achieve the outcomes. Teaching 
staff often say that their teaching is aimed at sophisticated 
and subtle learning outcomes, but rely on methods which do 
not provide opportunities for achieving these e.g. lectures/' 

The second reason why assessment became a major issue was that a focus on 
outcomes highlighted concerns about the nature and quality of assessment be- 
cause: 

• the relationship between the course objectives and what was currently 
assessed was not always clear, and assessment was often not treated as an 
integral part of a course. There was little evidence of an assessment 
strategy in many courses, and little sharing of information about this 
between staff; 

• tho traditional forms of assessment placed a heavy burden on staff when 
it came to marking, the increase in student numbers was making some of 
these methods impractical, if not impossible; 

• when intended outcomes were made explicit, it became clear that 
alternative methods of assessment were required and staff lacked 
experience or information on alternative approaches. 

1.4.4 Outcomes and Credits 

The project sought to create a link between learning outcomes and a system of 
credit accumulation and transfer. This was an essential step since it linked 
qualitative learning outcome statements to a quantitative system of credits 
which added up to a degree. The lirk could only be made if it was possible to 
describe the learning outcomes, to assess them, and to allocate a volume or 
value to them through credit rating. 

The project used the CNAA CATS credit structure which is widely established 
in higher education, and sought to define the credit value of learning outcomes 
using the CAT credit point system. 

The CATS scheme is particulariy appropriate for this puipose since it provides 
a system of credits and levels but does not seek to define a credit rigidly in 
terms of hours of study or notional study time, nor does it specify the subject 
content, or what must be achieved to gain a credit. However, the question of 
how to allocate credit to outcomes, which are of differing 'sizes' and 'levels', 
except by the use of actual or notional teaching/learning time, proved proble- 
matic. 




1.5 Project approach 



The design of the project was based on a model for developing descriptions of 
learning outcomes originally described by UDACE^ and shown in Figure 1.1, 

This proposed that there were three distinct kinds of 'users' of learning out- 
come descriptions, those who leam, those who provide learning programmes. 



Understanding Learning Outcomes. UDACE 1989 
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and those who pay or use the products of the process (which might be Govern- 
ment, individuals or employers) Each of these three has a distinct and legitim- 
ate perspective on the purposes of the education offered, its assessnrient and 
accreditation. It also proposed that an adequate description of outcomes should 
be able to recognise both those outcomes consciously planned in advance, and 
those which might arise unexpectedly as the programme progressed. Finally, 
the model proposed that there were three distinct kinds of assessment which 
might be applied to learning outcomes. In different circumstances the different 
users would wish to place differing degrees of emphasis on the 18 'cells' of the 
matrix. Some employers, for example, might be particulariy interested in crite- 
rion referenced, rather than personally or norm referenced outcomes, while 
those selecting for admission to further study have traditionally focussed on 
norm referenced approaches. 



Figure 1.1 -A Model of Learning Outcomes 

The project sought to provide an open and consultative structure which would 
permit the maximum discussion and dissemination of ideas ar^. materials. This 
was based on the premise that learning outcomes should b. everyday state- 
ments used and understood by staff, students and employers aUke. They there- 
fore needed to avoid specialist jargon and be owned and developed by those 
who would use them. The project was not intended to produce a set of "ideal" 
learning outcome statements written by experts, but rather to stimulate debate 
on what learning outcomes were, how they might be developed and used and 
assessed, and to produce some recommendations which would act as guidelines 
for others to use. 




Expected 
(objectives) 



Actual 
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1.6 The project process: an outline 



The project involved three groups of people in the definition of learning out- 
comes; academic staff, students and employers of graduates. The work in- 
cluded detailed development activity by small groups of academic staff who 
were subject "experts", followed by individual and group ccxisultation with 
other academic staff, with students and with employers of graduates. 

The project extended over a period of two years, involving 19 academic staff 
from 9 higher education institutions, who were primarily responsible for the 
preparation of some exemplar learning outcome statements and for investigat- 
ing the relationship between learning outcomes, credits and assessment. 

During the course of the project, over 50 meetings, workshops, and seminars 
were held, involving several hundred academic staff from other HE institutions. 



Employers 



Employer 
Seminar 1. 
Employer Visits 
(April 1990) 



Employer 
Seminars 
(Feb-May 1991) 



Employer 
Consultative 
Document 

Issued 
(May 1991) 



Academic Staff 

Institutional 
Consultation 
(November 1989) 

Selection of Institutions 
(December 1989) 

Subject Group Work 
(Jan-Dec 1990) 



Consultative Seminar 
(September 1990) 



Assessment Workshop 
(Sept 1990) 

Consultative Document 
What Can Graduates Do? 

to all HEIs 

(Jan 1991) 

Review Workshop 
(May 1991) 



Draft Report to Steering 
Committee (Oct 1991) 



Students 



Student 
Workshops 
(Nov-Mar 91) 

Assessment Pilots 
(Sep-Jun 1991) 



Figure 1.2 - The Project Process 
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The process of tlie project is summarised in Figure 1.2 and described in more 
detail in Appendix 1, In outline, it consisted of 8 stages: 

Stage 1 

All HE Institutions \^'ere invited to an initial consultative meeting to discuss the 
project aims and objectives and identify appropriate subjects for development 
work. The five subject groups chosen were Design, Engineering, English, Envi- 
ronmental Science, Social Science. 

Stage 2 

institutions were invited to bid to participate and nine institutions were selected 
by the Project Steering Committee. The individuals, seconded part-time from 
the institutions, represented a broad cross section of academic staff. None had a 
formal academic background in educational research or theory. Each subject 
group involved staff from more than one institution. 

Stage 3 

Staff were given three objectives for the twelve months work: 

to develop a model of learning outcomes for iheir subject area 

to develop a way of accrediting learning outcomes 

to pilot methods for the assessment of the learning outcomes 

The subject groups were encouraged to develop individual approaches to tlie 
description of learning outcomes, rather than to follow a single model. They 
were asked to consider the outcomes of degrees, rather than specific stages or 
levels of degree courses, and were provided with a range of background ma- 
terial and support though workshops and meetings. 

Stage 4 

A national seminar was held in September 1990 to provide an opportunity for 
academics from all HE Institutions to comment on the preliminary work. 93 
iMstitutional staff attended. Students were consulted through a series of struc- 
tured workshops with first and final year students in each of the participating 
institutions. Over 500 students were involved in these workshops. 

Stage 5 

Academic staff were asked to develop approaches to the assessment of a range 
of the learning outcomes identified, concentrating on those not covered by 
existing assessment processes. A workshop on assessment was provided for all 
project staff and a range of literature and materials was distributed. 

Stage 6 

After the national seminar, a consultative document. What can Graduates Do? 
was distributed to all HE Institutions, individuals known to be interested in the 
work and those attending workshops and conferences where project staff were 
present. Responses were received from over half the HE institutions in the UK. 



9 



Aims and Strategy 



Following a preliminary workshop with graduate employers, based on What 
Can Graduates Do? a revised version was produced for distribution to a broad 
range of employers. 138 responses were received from employers. 

Stage 7 

A residential workshop was held for project staff to review the outcomes of 
this work and the outcomes of the consultation. The responses to the student 
and employer consultations were combined with the outcomes of the academic 
staff work into a draft report to the project steering group ir the Autumn of 
1991. Following further informal consultation with the project steering group 
and project staff Ihe present report was completed in March 1992. 

1.7 Project Staff 

The project was directed by Sue Otter, UDACE Senior Development Officer 
for Accreditation and teaming Outcomes. In December 1990 she was ap- 
pointed Head of Credit Accumulation and Transfer at Nottingham Polytechnic. 
UDACE is grateful to the Polytechnic for agreeing to release her part-time to 
complete the project and the present report. 

Daffyd Thome was seconded to UDACE from the I^ndon Institute as Project 
Officer for 12 months. 

21 academic staff from 9 higher educaUon insfitutions contributed as members 
of the project's subject groups. They are listed in Appendix 2. 
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Learning Outcomes: an 
academic staff approach 

This chapter reviews some of the ideas and ways of describing 
learning outcomes which were used by the academic staff 
working in project groups. 

2.1 Initial premises 

The project began with a number of premises. 
Aim to develop methods and examples. 

The aim of the project was not so much to produce a definilive set of learning 
outcome descriptions as to identify effective and efficient ways of writing and 
structuring them and to produce guidance to assist others to describe the learn- 
ing outcomes of degree programmes for themselves. 

Begin with academic staff views 

It seemed that the most effective way of going about this was to ask a small 
group of academic staff (the subject groups) to develop some examples of 
learning outcomes for their subjects, and to use these as a basis for consult- 
ation, with other staff, students and employers. 

Work in small independent groups 

The five cross-institutional subject groups (see Appendices 1 and 2), were 
asked to work independently in developing the learning outcome statements for 
their subjects. They were not given a fixed set of guidelines on how to proceed, 
but they were asked to record and document their progress and to develop a set 
of descriptions which could be circulated for wider comment and discussion. 
There were two reasons for choosing to work in this way: 

• the project aimed to produce guidelines which could be used by other 
academic staff and it was generally felt that these would be best 
developed by small groups of practicing academic staff with direct 
experience of teaching and learning in HE in their subjects, rather than by 
external 'experts'; 

• it was felt that there might be several ways in which learning outcomes 
might be structured. Allowing the groups to work independently with few 
guidelines might generate novel and helpful approaches. 
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Examine previotis material 

The only examples of learning outcome statements developed for similar pur- 
poses readily available at the start of the project were provided from the USA 
(Alvemo College publications) and from the work of the Employment Depart- 
ment Standards Branch. 

The former were statements of a rather general nature, they looked like lists of 
desirable qualities but seemed not to have been derived through any analytical 
process. They also lacked sufficient detail to indicate the level of work and to 
allow ttiem to be assessed direcUy. 

The Employment Department work, carried cut in the development of occupa- 
tional standards for National Vocational Qualifications, were clearly derived 
from a more analytical process (functional analysis) and they were detailed 
enough to allow assessment, including performance criteria which were often 
very detailed. They were however at fairly low levels and described relatively 
simple tasks, they were thought not to contain sufficient detail of factual 
knowledge, nor of the more complex intellectual processes associated with 
higher education. 

Develop initial guidelines 

Initial consultations with the academic staff groups, suggested tliat learning 
outcome statements would need to: 

strike a balance between what graduates needed to know and 
what they needed to be able to do; 

include subject specific knowledge, as well as the methods, 
traditions and principles of subjects and disciplines; 

address the sorts of qualities sought by employers; 

be intelligible to students and employers, as weU as academic 
staff, while p 'esendng the essence of the outcome itself. 



2.2 First examples 

The early stages of the work showed that some outcomes were easy to de- 
scribe. These often related to factual knowledge and personal skiUs, 

the student will be able to explain the major mechanisms of 
mammalian thermoregulation^ 



the student will be able to present this information clearly and 
succinctly in a ten minute presentation to a group of staff and 
students. 



Others were much more difficult, 



the student will be able to present and sustain an academic argument. 



the ability to respond to alternative viewpoints/ the ability to speculate 
or present hypotheses. 
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There was a persistently expressed view that there were some outcomes of 
higher education which could not be described in these icvms, including the 
development of personal integrity, or the process of personal growth. 

Discussion and consultation showed that it was possible to imp' '^e descrip- 
tions of some of the previously implicit values thought to be impaired through 
higher education. Other outcomes, notably those concerned with professional 
ethics and moral values, were clarified through discussion witli staff and stu- 
dentSt although they were not always well captured in words. 

2.3 Consultation 

At the mid point of the project the draft outcome descriptions of the subject 
groups were collated into a publication What Can Graduates Do? which was 
circulated to all higher education institutions and others for comment. The re- 
sponses addressed both the specific examples prepared by the groups, and the 
general principle of an outcome led approach to higher education. The range of 
comments is reflected in the following quotations. 

"It is true that outcome descriptions cannot be sufficiently 
sensitive to adequately reflect the full range of knowledge, 
skills and understanding which a degree represents, or the 
complexity of the relationship between the parts, but that does 
not mean we should not try, nor that we cannot evolve an 
ever closer approximation to a complete description," 

'The definition of learning outcomes is quite acceptable. 
However, to distinguish between outcomes as statements 
rather than as a set of constructs like knowledge, competence 
or skill, seems to misunderstand competence statements. 
Similarly, the contrast made between learning outcomes and 
objectives does not hold up, and reveals the authors' lack of 
expertise in this area. Objectives do not set out to describe 
what a course will do but define what students can do as a 
result of participating in the course." 

"Any clearly stated and understandable set of words w,,.^h 
can describe more precisely the con^plexVange of learning 
experiences and identifiable outcomes would be welcomed by 
all parties. However outcome descriptions alone may be 
insufficient to accurately describe the iterative process which 
governs the creative and intuitive harnessing of ideas into 
recognisable solutions" 

'What is common to all courses of higher education is that 
they are not about acquiring particular knowledge or skills 
but involve a very significant process of personal growth, but 
this is wholly ignored in your document." 

"Talking about learning outcomes also de-empliasizes the 
variety of higher-level outcomes that different students 
achieve at a personal level which are nevertheless compatible 
with successful performance at a lower level in their formal 
examinations and other assessments,'* 
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2.4 Describing Learning Outcomes - four 
approaches 

Four different approaches to describing learning outcomes were explored dur- 
ing the project. Some were developed and used by the groups and others are 
derived from work outside this project The four approaches were based on: 



Objectives 
Subject knowledge 

Discipline 

Competence 



the stated intentions of the course. 

the knowledge content commonly 

identified in syllabuses or course documentation. 

the notion of a discipline as a culture and value 
system to which the graduate is admitted. 

what a graduate can do as a result of the degree 
programme, including the narrower notion of 
occupational competence. 



2A.1 Objectives 
Bxiisting course materials 

Since CNAA course validation processes require the preparation of more de- 
tailed documentation than is common in other HE institutions, the subject 
groups were designed so that each would have at least one member from such 
an institution, with access to existing documentation prepared for course vali- 
dation. All the groups were encouraged to provide and circulate this informa- 
tion since it contained details of aims, objectives and assessment methods 
which it was thought would be helpful in developing learning outcomes. 

Although some of this information was circulated, only one group (Social 
Science) actually used course documents as a basis for discussion. One other 
group (Engineering) used course documentation to determine the number of 
hours, and the assessment weighting of each of the course components from 
each of the three institutions. 

The observation that these documents were not as useful as ha:! been expected 
was initially surprising. However, reviewing the course documents now, 
against the outcomes described by the groups, it is clear that many of the aims 
and objectives presented are considerably less detailed and comprehensive than 
the learning outcome statements. This does not suggest that learning outcomes 
are in some way superior to objectives, nor even that they are fundamentally 
different. Rather, it suggests that existing approaches to writing course aims 
and objectives can be improved, and that thinking about the course in terms of 
outcomes can be helpful. It might also suggest that the preparation of aims and 
objectives by course teams and committees has sometimes carried out more to 
meet tiie requirements of validation, than to serve the purpose of developing 
teaching, learning and assessment strategies. Subsequent comment from HE in- 
stitutions on the consultative document What Can Graduates Do? provided 
conflicting responses on this issue. There were suggestions both tiiat aims and 
objectives were indistinguishable from learning outcomes and that they were 
viewed simply as a formal requirement for validation. 
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Bloom's taxonomy 

One of the background documents which was cu*culated to group members at 
the start of the project was a summary of Blooms Taxonomy of Educational 
Objectives (1964) ^. This was also referred to by many correspondents. Al- 
though the title and nature of this work were familiar to some of the group, few 
were familiar with its content, although most had played some role in develo- 
ping and writing objectives for the courses they taught. 

The Taxonomy was initially useful in providing a standardised language (a set 
of action verbs in a hierarchy). These could be included in the outcome state- 
ments and their meaning was clearly understood by everyone. For example, 
those outcomes which had previously read "the student will have a knowledge 
of, could now be rewritten as the "the student will know, understand and be 
able to ^ply", or "the student will be able to analyse" (by implication this 
included, know, understand, analyse, and synthesise). Group members were en- 
couraged to use the Bloom language to clarify their learning outcome state- 
ments. 

Some verbs were particularly confusing, although some of this confusion did 
not become apparent until the groups were asked to look at ways of assessing 
the outcomes. It then became clear that they were too vague, and did not pro- 
vide enough detail to allow a judgment as to whether a student had achieved 
them or not. "Know" and "understand" were especially difficult, as were "have 
a knowledge of, and "be acquainted with". For example, the Design Group 
identified the following learning outcome: 

understand aspects of the taw as U applies to designers 

The problem with this statement centres on the use of the word "understand", 
which does not make clear whether the student is expected to know in detail, or 
know in general, or know and be able to apply. In discussion it became clear 
that the outcome was in fact as much concerned with knowing how to find out 
about specific aspects of the law, as with knowing and understanding its iiTipli- 
cations. The word "understand" was understood by the course staff, but poten- 
tially misleading to students and others. 

Other problem expressions included "have a grasp of, and "develop a working 
knowledge of. Qarification of these expressions was achieved through dis- 
cussion with staff in the subject groups and with students. Discussions with the 
latter group proved to be productive and helpful, although it was clear that 
however well the staff felt that the outcomes were described, the students con- 
tinued to seek further clarification. 

The groups developed various ways of clarifying the precise meaning of the 
learning outcomes they produced for What Can Graduates Do?, Some used the 
work on assessment to provide details of exactly what a student was expected 
to do, others used the teaching methods and processes to clarify what they 
required. 



3 Sec Hotel. 



ERIC 16 



Learning Outcomes: an academic staff approach 



In some cases, the description of the teaching/learning process helped to clarify 
the outcome itself. For example the Social Science Group produced the follow- 
ing: 



Knowledge of the present day functioning of major social institutions 



There were various ways in which this could be developed more precisely, 
these involved using work experience to collect information on a series of pre- 
viously agreed questions, developing case studies as a class exercise, and re- 
searching information for project work. Qarifying the process to which the 
outcome related helped to clarify exacUy what the student was being expected 
to be able to do. Students were cleariy expected to be able to find and analyse 
information rather than simply read a set text. The outcome also called for the 
ability to analyse, evaluate and compare the functioning, and these questions 
could be examined with reference to social issues in general, or in particular. A 
student might also be required to speculate about future trends or conse- 
quences. TTie group felt that the nature of assessment and the criteria for judg- 
ing student achievement of the outcome would vary between, and possibly 
within an institution, and that in general the outcomes should be discussed with 
groups of students in oider both to clarify the meaning and agree cntena for 
assessment. 

An alternative application for the Bloom Taxonomy was suggested by some 
later work on credits and levels, where it was suggested that the Bloom's ac- 
tion veri3S could be used to describe learning outcomes at various levels. Level 
one for example, would include knowledge and comprehension. Level two ana- 
lysis and synthesis and so on. The suggestion was dismissed by the groups as 
being too simplistic: analysis, for example, was beUeved to be just as much a 
feature of Level 1 as it was of Level 3. 



Carter's taxonomy 

A second document was Carter's Taxonomy of Objectives for Professional 
Education (1985)"^. This proved to be very valuable to the Engineering group, 
whose first draft of learning outcomes had been sharply criticised for its heavy 
emphasis on factual knowledge. 

The Engineering group initially presented their learning outcome statements in 
a series of headings. One of these was Processes and Materials, wiUiin which 
was Manufacturing Methods, including the following outcomes: 



A student would be expected to: 

- select and reject manufacturing processes to produce a specified 
component economically, 

- plan the sequence of the processes required for the manufacture of 
the component, 

- propose tests and quantify the results for the quality and reliabiluy 
of the manufactured components, 

- use statistical methods for quality assurance, 

■ program for computer aided manufacture and materials handling. 



Sec Note 1. 
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The Engineering group found the taxonomy helpful and produced a second 
draft using the Carter structure, which set these very detailed outcomes in a 
broader framework, and added a dimension of personal qualities to the first 
drafts. Examples of the learning outcomes produced are described below. 

Menial Skills 

Identify Relationships between pieces of information 

' analyse forces and stresses within an engineering mechanism or 

structure, 

' analyse transducet specifications to ensure satitfactory automated 
systems response, 

' calculate tolerances on components. 



Personal Qualities 
Attitudes and Values 

Demonstrate an awareness of strengths and weaknesses 
- set and meet realistic goals for personal performance 
" seek assistance in remedying areas of weakness 



The structure diu however have some drawbacks. The resulting detailed state- 
ments were repetitive, and some outcomes could be included under several 
headings. The structure might be more helpful used less rigidly, perhaps as a 
scries of prompt questions or, as the Engineering group used it, as a basis for 
reviewing a set of learning outcomes to ensure that all aspects were included 

2.4.2 Subject Knowledge 

A second approach was to examine what was being taught on existing courses, 
and to attempt to convcn this into outcome siaiem-ents. This inevitably began 
v/ith a description of subject knowledge. However there was debate about the 
degree to which the acquisition of knowledge was itself the outcome, rather 
than the implicit cognitive skills which individuals developed through acquiring 
it. Groups varied in their approaches to this question. Some began by grouping 
and listing factual knowledge under a series of headings, while others began by 
looking for cognitive and affective skills. An eariy draft produced by the En- 
gineering group listed factual infomiation as learning outcomes and was exten- 
sively criticised at the Consultative Seminar held in September 1990. 

For example, some typical learning outcomes taken from the paper produced 
by the Engineering group for the consultative seminar included: 



- calculate energy values and forces present when electromagnetic 
induction is produced by interacting electric and mar; le tic fields 

- use complex numbers and phase diagrams to calcula te values when 
using AC circuits 

- design an amplifier circuit and be aware of its characteristic- 



The document contained several hundred statements of this kind and 
was generally thought to be ''too factuaV\ Several commentators at 
the consultative seminar, suggested that it was necessary to integrate 
the practical skills and competences with the knowledge, rather than 
simply listing them separately. 



The group also developed a series of diagrams to relate areas of knowledge. 
Figure 2.1 is an example drawn from the eariy work of the group. 
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The Environmental Science group on the other hand tried to identify the under- 
lying cognitive skills using past examination papers. At the same consultative 
seminar, the Environmental Science learning outcomes were criticised for 
being too general (they could apply to any graduate) and for not containing 
enough specific factual detail about Environmental Science. 

For example, the document listed outcomes under nine headings based on types 
of student activity. Tne outcomes included: 

FieUwork 

- use scientific equipment correctly and appropriately 

- plan and carry O'it a sampling strategy in the environment 

- record filed observations as accurately as possible 

- appreciate the difficulties of operating equipment in the field and 
the inaccuracies which may result, etc. 

The debate about the relationship between knowledge, as lists of factual infor- 
mation, and cognitive skills, posed some interesting questions about the extent 
to which students understood the implicit cognitive skills they were expected to 
develop. It was not clear from the subsequent consultatiion that these were the 
skills which students themselves expected to develop as a result of a degree 
course. It was equally not clear that courses were designed to develop, or give 
students opportunities to practice these skills. This point became more apparent 
when methods of assessing outcomes were considered later. 

It appeared from the responses to the consultative seminar, that leaming out- 
comes needed to fall somewhere between lists of factual information and lists 
of cognitive skills removed from a subject context. Both the Engineering and 
tiie Environmental Science groups subsequenUy developed their leaming out- 
comes in a structure which separated factual knowledge from practical and per- 
sonal skills. The Engineering group opted to use the Carter Taxonomy for this 
purpose, and the Environmental Science group followed a similar stmcture. 
Both structures proved to be problematic when the groups were asked to think 
of ways of assessing the outcomes. It quickly became clear that the personal 
skills and competences were not separable from the context of the subject, and 
were often better used as criteria for the assessment of Uie factual knowledge. 
For example a leaming outcome might describe an area of knowledge which a 
student was expected to "know, and understand". When asked "how will you 
know that the student knows/understands this", the response was almost invari- 
ably that s/he was required to communicate this information (orally or in writ- 
ing), or apply Uiis information (to the solution of a problem), or use it as 
evidence in the development of an argument or hypotheses, or to illustrate a 
point in debate. In practice, therefore, a range of personal communication skills 
were being assessed alongside the specified knowledge. 

Further discussion raised some interesting questions about why a student was 
required to know certain things, and how these related to the course as a whole. 
There were clearly several categories and priorities, some things were "vital to 
know, and fundamental to the understanding of the subject", some "were use- 
ful" and some simply "interesting". 

The responses to the consultative document continued the debate about the 
place of knowledge. Some suggested that: 
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'*the capacities our graduates develop should be related more 
specifically to a wide range of historical texts". 

and others: 

'What bothered me the most about the English section - and 
indeed about a focus on learning outcomesi competence 
generally - is the almost complete absence of talk about 
knowledge. This seems especially inappropriate with respect 
to teaching and learning in HE (and indeed there is 
correspondingly very little attention paid to teaching 

too) critical thinking, interaction, personal and social 

outcomes are of course important in any course or degree in 
English, but surely so is having studied a (some) range of 
literary texts, and having attempted to understand them in 
their own terms and in the cultural/ historical circumstances of 
their production". 

A further scries of responses suggested that the statements should attempt to 
describe outcomes which were at a more generic level and applicable to a 
range of subjects/courses. Specific detail could then be added to each outcome 
using an illustrative statement. This idea found favour with several correspond- 
ents: 

'This is a highly desirable system. It is also consistent with 
current thinking about the nature of transferable cognitive 
abilities and. skills. I would go so far as to argue that it is 
when we have learned to generalise sufficiently to take a 
learning outcome away from one specific context, that we are 
at the heart of the ability which we look to see learners 
acquiring". 

2«4«3 Disciplines 

A third approach started not from the content, or ihe course objectives, but 
from the notion of a discipline. This asked the question "what is a Social 
Scientist or an English graduate?" 

Several groups (notably Social Science and English) debated the nature and 
elhos of their subjects, and what they understood by 'disciplines', and used 
these debates to inform the writing of their learning outcomes. These groups 
clearly felt that knowledge was not just a "body of irrevocable truths" (Schwab 
1962) but was concerned with patterns of procedure, methods of enquiry, and 
ways of distinguishing relevant and irrelevant information, and that there were 
subject specific differences which could be probed and described in more de- 
tail. There was an entirely different approach to factual knowledge, and the 
amount of 'essential/ need to know/cannot graduate without knowing tliis' in- 
formation was much less. What proved to be much more difficult was arriving 
at a clear definition of the underpinning structure, the methods of enquiry and 
the procedures. These things were clearly implicit in the discussion between 
subject specialists, but often difficult to explain, in other than superficial terms, 
to a non specialist. 

The composition of the groups made the exploration of some of these ques- 
tions difficult, since there was not always a consensus about the terms 'discipli- 
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ne* and 'subject*. EHscussion did however expose other useful points related to 
learning outcomes and course structure. Some of the degree courses repre- 
sented were single subject courses, while others were modular and interdiscipli- 
nary. There were therefore tensions and conflicts within some of the groups. 
This was most pronounced in the English group which included courses in 
English, in Communication Studies and a combined degree in English and His- 
tory. Where the degree was a linear course, concerned with the study in depth 
of a single subject, the learning outcomes seemed necessarily to include ones 
which related to the structure and methods of enquiry of that subject. This was 
less cleariy established in some of the modular degi^ees where several different 
subjeas were studied together. This particular group faced a further problem in 
that the ethos and methods of enquiry in English were quite different from 
those in Communication Studies. This resulted in an uneasy compromise which 
was reflected in the diversity of responses to the consultative document, one 
described the learning outcomes in English as * mythic constructs', while others 
were very positive, 

"The contributors have striven to view learning outcomes in 
an objective, almost scientific spirit, and the results are most 
useful and impressive. The distinction made between higher 
level outcomes and median outcomes is well conceived, as is 
the outline of learned competency", 

"The university study of English Literature involves many of 
the skills enumerated, though not particularly the forensic and 
managerial skills mentioned in points 28-33 (communicative 
competences) and 36 (networking and finding information)," 

The idea of developing outcome statements from an understanding of the disci- 
pline was used most successfully by the Social Science group, who sought to 
describe generic outcomes which could be used across different subjects (psy- 
chology, sociology), rather than attempting to describe outcomes for each sub- 
ject. This group was one of the few which chose to work through course 
documentation, and an early draft of materials described the following: 

There are six elements to the first year programme. Our method in 
considering them was to avoid specifying curriculum content unless it 
was clear that the learning outcome aimed for, orspecifxally 
demanded, knowledge or definite material 

We found that almost always the material with which the course was 
ostensibly concerned was a vehicle aimed at introducing or applying 
broader theoretical perspectives.The real learning outcomes aimed at 
therefore had to do with mastering the essence of theories, 
distinguishing one theory from another, understanding the 
parameters underlying the whole range of theories and, most 
importantly being able to apply different perspectives dispassionately 
to social phenomena. It was apparent that implicit in the knowledge 
based outcomes were a variety of cognitive skills: 

- factual knowledge; 

- understanding (ie knowing why a certain question is relevant to the 
discipline etc); 

- tools of thought (vocabulary); 

" awareness of the dimensions of debate; 
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- awareness of the possible fields of application of the discipline^ 
Af mg with these cognitive skills came personal qualities involving: 
' a high valuation of logic; 

• tolerance of the complete relativity of both social phenomena and 
theory; 

• a readiness to set aside presuppositions and prejudices about the 
**real world**; 

' a readiness to see ^^taken-for-granted facts** as actually 
problematical. 



There are alternative views on the structure of knowledge which rely on the 
relationship between knowledge and the use of knowledge, and are derived 
from studies of professional and post experience education. Some of these were 
proposed in responses to consultation, although none was applied during the 
project,^ 

In the end some groups used factual knowledge as one of a series of headings 
in describing their learning outcomes, (Engineering and Environmental 
Science) while others used knowledge processing or cognitive skills as learning 
outcomes and factual knowledge to put the cognitive skills into a subject spe- 
cific context (English and Social Science). 

2*4.4 Competence 

The fourtJi approach focused around the notion of competence, asking the 
question "what can a graduate do?" This stimulated extensive and heated de- 
bate with many correspondents, mainly related to the use of the term by the 
National Council for Vocational Qualifications and the Employment Depart- 
ment in relation to the then new national vocational qualifications. There was 
widespread confusion and ignorance of current developments, and many people 
were unsure about the distinction between competence as a broad set of abil- 
ities which could be applied in a range of settings, and occupational com- 
petence, which was being developed as a much narrower description of what a 
person was required to be able to "or employment in a specific occupation. 
There was concern tliat the notion of occupational competence was inappropri- 
ate in higher education and a number of correspondents were concerned that 
the project might be a "covert attempt to take higher education down the same 
route". 

Two major issues caused concern: 

• vocationalism, the fear that important dimensions of higher education, 
which relate to the development of the person and to the transmission of 
cultural and social values, would be lost in the pursuit of short term, woiic 
related, value systems, 

• reductionism: that the analytical techniques which break learning into 
smaller pieces inevitably lead to the neglect of those qualities which we 
understand as 'subject', 'profession' and 'discipline'. 



5 Sec Note L 



ERLC 3^ 



The Project Process 



"Assessing outcomes w a very useful device, and it is 
important to publicise and maximise good practice. But it is 
also important not to overlook the process of education, and 
its role in personal development, or in the development of 
personal attributes and qualities that are not as easily defined 
as are some skills. It would be unfortunate if a focus in 
outcomes unwittingly led to a reductionist view of the 
opportunities (and dare one say enjoyment) which should be 
inherent in the student's educational experience/' 

Since it is clear that, at some level, most people believe that being a graduate 
represents some kind of broad competence, it was clearly important to seek 
ways of distinguishing a broad notion of competence from occupational com- 
petence and to search for acceptable ways of describing competence in relation 
to subjects and academic study, as well as in relation to employment. This was 
one of the reasons for involving graduate employers in the work, since it was 
felt that they could help in the definition of a broad set of competences which 
were related to woric but which were not occupationally specific,^ 

Problems 

Ail the groups experienced problems in developing a rounded and detailed de- 
scription of graduate competences. Apart from the Design group, which ana- 
lysed the design process in detail in order to identify its learning outcomes, 
none attempted a systematic approach to describing a complete set of com- 
petences. 

Several problems were identified in discussion and responses to consultation: 

• it was relatively easy to identify broad competences in general terms, but 
some people thought that these were * trite' although oUiers thought it was 
important to include them to ensure that the obvious was not overiooked. 
This view was supported by the responses to the consultative document, 
several of which suggested that ^'listing the bread and butter" was an 
important first step, 

• even if the competences couid be described there were difficulties with 
assessment, and the groups were generally vague about standards - what 
constituted good, or not good enough performance of a competence. This 
was in part related to a confusion between competences, which could be 
learned and developed, and * fixed' personality traits. The example of a 
first class graduate with poor oral presentation skills was quoted several 
times, by correspondents who believed that the ability to communicate 
was a fixed trait rather than a developable competence. 

• some people believed Uiat "these sorts of things should be developed in 
school", and that they were "not the responsibility of higher education". 

• some people argued that competences were of secondary importance and 
that although it was desirable to have them, a student would not fail if 
s/he did not. Other responses to consultation suggested quite the reverse 



6 This issue will now be partially addressed in the development of general National 
Vocational Qualifications 
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and described graduate competences as "key to the whole concept of 
learning outcomes". 

Group approaches 

The groups approached the inclusion of competences in various ways: 

The Design Group analysed the design process in order to identify the com- 
petences and the specific knowledge they required. They were carefiil to avoid 
specif>ang a particular process or sequence, and were helped in doing this by 
the inclusion of two and three-dimensional specialists, and a specialist careers 
adviser. 

The Environmental Science Group looked at the various learning activities, 
tutorials, field woilc etc. which students were required to participate in, and 
identified the competences they were expected to develop through them. 

The Sociai Science Group undertook some functional analysis v/ork with a 
group of "professionals". This was useful, not because it provided an exhaus- 
tive list of competences, but because it provoked a debate on values and ethics, 
and identified the need to identify and work within the values of individuals 
and organisations as an essential competence for social workers. 

The Engineering Group used the concept of a *good graduate* to develop a 
set of competences for inclusion in their learning outcomes at an early stage of 
the project. Most of these were lost from the drafts of learning outcomes de- 
veloped initially but some were reinstated later. 

The English Group developed a competence model for communication 
studies. Figure 2.2 is an early example of such a model 

An alternative approach not explored directly in the project was suggested by 
the work of Klemp (1974) and Elliott (1990), Klemp used an empirical ap- 
proach to describing the competences developed in post secondary education 
analysing the distinctions between exceptional and average performers. ^ 

Functional Analyc^is 

In the development of techniques for outcome description in vocational qualifi- 
cations the Department of Employment has evolved a form of functional ana- 
lysis which offers a systematic hierarchical approach to competence. This 
approach was relatively new and unfamiliar to the project groups, they were 
therefore offered a woricshop to introduce them to the methods and ideas. 

Functional analysis is a systematic and logical approach to describing an occu- 
pational role, in order to produce 'standards' which are used for the award of 
vocationally specific qualifications. It begins with the definition of a 'key pur- 
pose' for an occupational role, which is then disaggregated first into 'units', 
and then to 'elements', by asking the question, ''what needs to happen for this 
to be achieved?'' (Mansfield 1989). This develop)s a hierarchical structure at the 
bottom of which each element is accompanied by a set of performance criteria 
for assessment through which an iiidividual can be recognised as competent. 



7 See Note 1. 
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The functional analysis workshops provoked a mixture of curiosity and hos- 
tility. Apart from the concern with vocationalism and reductionism there were 
three principal concerns: 

• the direct link between the current applications of functional analysis and 
specific occupations. This was problematic because students in all the 
subjects subsequently took up employment in a wide range of 
occupations, it was therefore in some cases difficult (Social Science), and 
in others impossible (English) to define an occupational role: and none of 
the academic staff saw their courses, or their learning outcomes, as direct 
vocational preparation. The fact that a functional approach seeks to be all 
embracing makes it difficult to recognise such a diversity of outcomes. 

• the narrowness of the approach. Here respondents felt that it produced 
detail of the "here and now" of the occupational role while they were 
concerned to provide their students with a more generic set of intellectual 
processing skills which would enable them to respond to the less 
predictable needs of the future. 

• the apparent difficulty of accommodating questions of values, and the 
dilemmas, contradictions and constraints which were present in all 
"real-life" situations. Some respondents felt that there seemed to be no 
place in functional analysis for the kind of "mental balancing act", which 
resulted in good decision making and judgment at professional levels. 

The workshop provided a valuable forum for discussion and development, one 
group (Social Science), subsequently held a functional analysis workshop with 
a group of professionals from the social sciences. This produced two contrast- 
ing key purposes. 

To provide high quality infortnation about the social composition of 
society and about social trends found within society y so as to enable 
government policy formation and business decision making to be both 
effective and legitimised. 



To appraise social situations, events, institutions and individual social 
behaviours, in terms of its sources in interpersonal or societal factors. 

The contrasting nature of the statements reflected the faa that the group did not 
perceive Social Science as a single subject, and had varying academic and pro- 
fessional experience. Tney also observed that the nature of social science itself, 
particularly the contestability of all social judgments made it difficult and per- 
haps undesirable to define a single key purpose. However the exercise ol at- 
tempting to develop one had been a useful experience. This led the group to 
attempt to define a set of generic learning outcomes which could be applied 
across all the subjects which made up social sciences. The group saw these as a 
basis for individual institutions to develop in their own way, reflecting their 
separate subject specialisms. They chose to illustrate the generic outcomes with 
reference to specific subjects, in order to demonstrate their applicability. 
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Knowledge of the elaborations and underpinnings of the main 
perspectives within the relevant social scientific and other disciplines, 
and the ability to use this knowledge to perceive the underlying 
dimensions that differentiate theories and explanations. 

Taking psychology as an example; in becoming aware of the 
spectrum of psychological approaches (socio-biology, cognitive 
psychology, psychoanalysis, behaviourism etc), a student will partly be 
able to understand each of the theories by relating them to the others. 
In the process the student will attain the understanding of the 
parameters that distinguish them. The same sensitivity to significant 
differences of perspective will operate throughout the social sciences, 
so that in sociology, a student would be expected to be able, for 
example, to identify, and then assess the consistency of a functionalist 
account of deviant behaviour; and thereafter appraise that account 
utilising both the strengths and weaknesses of functionalist accounts 
in general, and knowledge of other/rival cognate perspectives. 

The responses to this approach from the consultative exercise were generally 
positive, 

"These are comprehensive and interesting, the examples are 
sensible." 

although the key statements were extensively criticised. 

"I would delete the words "government" and "business" from 
social science key statement one. It seems unduly restrictive, if 
perhaps politic, to see social science as serving these two 
masters only." 

None of the other groups chose to follow this pattern of working. The Engin- 
eering group did carry out some functional analysis work, but this produced a 
very broad set of statements, which the group found difficult to relate to the 
separate engineering subjects, and and they did not pursue the idea further. 

The remaining groups looked at writing key purposes or statements, but they 
almost invariably produced an aim or objectives for their courses rather than a 
key statement. This was helpful to them, often not because it generated a useful 
statement but because it focused the groups attention on defining what their 
courses were for, and it helped to expose some of the implicit aims of the 
members. In practice, the learning outcomes subsequently described by the 
groups did not relate to the key purpose, nor did the groups use the statement 
as a check on the inclusiveness of their outcome statements. 

In general the consu'*ative exercise suggested that some people found the key 
statements useful, while others found they distracted from the outcomes. 

"/ did not find the key statements useful. I suspect that much 
time can be spent trying to obtain agreement on a statement 
which can then mean very different things, even to the parties 
to the agreement. It is when we sit down and try to work out 
what we are talking about, in terms of level 4 (that is the 
learning outcomes), that definition really emerges and can be 
honed to precision." 
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It appears that they may be a useful vehicle to stimulate discussion, but may 
not be an essential step. 

The functional analysis documentation did provide useful ideas on the writing 
of outcome statements, for example it was helpful to write leaxning outcomes 
as short statements contai'iing a subject, object and verb, and to accompany 
these with a series of short phrases which qualified the outcc«ne and provided 
information relating to the level and the standards which would be sought in 
assessment 

2.5 Conclusions: a staff approach 

The learning outcomes developed by the groups of academic staff were arrived 
at through a process of discussion and consensus, rather than by a single 
method of structured analysis. This does not suggest that analytical approaches 
do not exist, but that the groups did not choose, or find, single models appro- 
priate for their puiposes. They did establish that a variety of methods can com- 
plement each other. Useful approaches included the taxonomies of Carter and 
Bloom, the knowledge structures of Eraut and Klemp, the Kolb Learning Styles 
Inventory, and functional analysis. The language of describing outcomes was 
particularly helped by reference to Bloom, but the most important factor in 
achieving clarity and precision was the requirement to assess the outcomes. 

The work on competences and learning outcomes was intended to be explora- 
tory testing a variety of approaches, and did not seek to provide complete 
single models for any individual degree programme. Neither did it seek to pro- 
duce a complete set of generic graduate, or subject specific outcomes. It did 
help to distinguish the notion of graduate competence from the definition of 
occupational or vocationally related competence, and this was a useful step. 
Subsequent work on assessment and accreditation showed that it was not help- 
ful to separate the description of graduate competence from the description of 
subject knowledge. In some respects this accords with the approach taken by 
NCVQ in developing occupational standards, although it is clear that knowl- 
edge in HE is more than merely 'underpinning', and the rationale for the inclu- 
sion of subject knowledge certainly not simply one of possible vocational 
relevance. These findings contributed subsequentiy to discussion on the co- 
herence of degree awards and it seems likely tiiat the question of coherence 
and balance needed to be debated at an earlier stage. 

Various approaches to writing and structuring description of learning outcomes 
emerged from tiie first stage of tiie work of the subject groups. The Design 
group found tiiat the design process was tiie most convenient way of grouping 
sets of learning outcomes developed by brainstorming and consultation. The 
Engineering group used a structure which had separate categories for knowl- 
edge, skills and competences. Environmental Science followed the same route. 
Botii English and Social Science separated intellectual skills (critical thinking) 
from personal skills and from other so called competences. 

The following stages of tiie work suggested that it was not helpful to separate 
out learning outcomes under headings like knowledge, skill and competence., 
since, although all the groups began by separatiing learning outcomes concerned 
with knowledge from those concerned with competence (Engineering and Envi- 
ronmental Science arc both good examples), tiie separation eventually led to 
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problems when ways of assessing learning outcomes were considered. It was 
impractical to assess competence separately from knowledge, since knowledge 
acted as a vehicle through which the competence was demonstrated. Similariy 
the assessment of knowledge invariably required the demonstration of some 
sort of COTipetence in communication, however defined. 

Later work suggested that an earlier consideration of the question of the 
relative value of the outcomes would have been useful, as would a more de- 
tailed consideration of questions of balance and coherence. 

Despite the difficulties and anxieties outlined above in the last meeting of the 
groups prior to the publication of the consultative document (What Can Grad- 
uates Do?) an attempt to systematise the learning outcomes was made by intro- 
ducing a series of levels. These were not levels of attainment, but were 
intended to be stages in the description of a learning outcome. The structure 
proposed that the first stage was the writing of a key statement or purpose, 
which might combine the notion of a good graduate, and the underlying ethos 
and values of the subject. The second stage broke this down in to three broad 
categories, skills, knowledge and personal qualities or competences, the third 
stage separated these into major activities or areas of the subject, the fourth 
level gave rise to learning outcome statements and the fifth to statements de- 
scribing what a student must do to demonstrate the learning outcome. 
Responses to the consultation suggested that people found this useful, although 
some found it too complicated and proposed a reduction in the number of *le- 
vels'. 
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Learning Outcomes: 
student and employer views 

This chapter describes the consultation with students and 
graduate employers. It provides a brief analysis of the 
processes and findings and draws conclusions from these, 

3.1 Collecting student views 

The student views on learning outcomes were collected through a series of 
structured workshops^. Students woriced in groups to prxxluce descriptions of 
expected outcomes (in the case of first year students), and outcomes achieved 
(in the case of final year students). The workshops generated an enormous 
amount of information, and demonstrated the differences between student and 
staff perceptions of higher education, but this woik did not contribute directly 
to developing ways of writing learning outcome statements. 

3*1.1 Evaluating the workshops 

The students welcomed the novel and helpful opportunity to participate in the 
discussion of what they were expected to achieve. The outcome approach also 
enabled tfiem to play an active part in the discussion of assessment, and this, in 
itself, helped them to clarify what they were expected to achieve. 

The workshops provided an effective structure for exploring student views but 
demonstrated that students could not, without extensive support and discussion, 
write their own learning outcomes ab initio. They were, however, able to de- 
scribe vocational and personal outcomes in greater detail than the staff. These 
outcomes agreed closely with the responses made by employers. Overall, stu- 
dents and exployers appeared to have more in common in their expectations of 
higher education than either did with academic staff. 

Students described relatively few of the outcomes listed by the academic staff 
groups. The fact that Uiey described neither factual outcomes, nor generic com- 
petences and cognitive skills suggests that they may not be aware of, or under- 
stand with sufficient clarity, some of the implicit expectations of academic 
staff, but they do have a much clearer perception of the qualities sought by 
graduate employers. 
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3,1,2 Analysing the student responses 

The student descriptions of leaming outcomes were analysed against the educa- 
tional orientation categories described by Taylor (Marten et al 1984) This 
showed 46 % of the outcomes were in the personal and social category, 39% in 
the academic category, and 15 % in the vocational. 

In all, 45 different "types" of leaming outcome were identified. These ranged 
from acquiring analytical skills and critical thinking, to developing IT and li- 
brary skills and increasing self esteem and independence. The most frequently 
identified outcomes were: 



enhanced career options 



increased independence 



increased self esteem 



acquire a broad base of skillsl techniques 



increased knowledge 



cirical thinking 

Few of the student outcomes related to specific academic subjects. 

These findings agree with a CBI Gallup poll of students at the point of entry to 
higher education (CBI 1991), in which 64% of the students interviewed said 
that they expected higher education to develop their personal and social skills. 
It is clear that students expectations from higher education differ significantly 
from those of staff. As Entwistle suggests^*^; 

"Tutors often seem to expect students to have the same 
interest in academic goals as themselves. In fact, students 
vary very markedly in their goals and purposes 

..students enter higher education with vague, perhaps 
unrealistic, but genuinely high expectations of the learning 
experience" 

but as Percy and Salter point out: 

"Ideas of academic standards, pursuit of excellence and 
scholarship are not among the constructs through which 
students evaluate the process of higher education}^ 



9 The 45 types of outcome arc listed in Appendix 6 together with illustrative examples. 

10 Entwistle 1990 



11 



Percy and Salter 1976. 



The Projea Pnx:ess 



3.1.3 Student views: overall findings 

The date provided from the workshops proved to be useful in two ways. Firstly 
it suggested that students were well able (more able than their tutors expected) 
to articulate their expeaations of their degree courses. Secondly, the workshop 
structure developed for this project was simple and robust and generated dis- 
cussion and opinions which could readily be used as part of a student consult- 
ative process. The data itself contributed to the development of a 'model' 
which was used as an intermediate step in relating learning outcomes to credit 
structures. 

3.2 Collecting employer views 

3.2.1 Methods of consultation 

Consultation with employers was achieved in several ways, firstly by a large 
circulation of materials to members of the Association of Graduate Recruiters, 
secondly through a smaller circulation of employers of Design graduates. There 
were also interviews and discussions with individual employers, and several 
employer representatives tc)ok part in some of the academic staff discussion 
sessions. 

12 

An initial consultative workshop, involving a small group of employers was 
used to edit What Can Graduates Do? to produce a short version which was 
sent to 500 graduate employers. 

3.2.2 The general and the occupationally 
specific 

Workshop participants were clear that many employers were looking for a 
generic set of "graduate" rather than occupational competences. They were not 
concerned with the subject and often the class of the degree. Some employers 
used sophisticated selection techniques to identify the competences they sought, 
while otiiers tended to rely on more general and impressionistic methods. It 
was therefore decided to edit the consultative document, presenting the infor- 
mation in two sections, the first described a set of 'common learning out- 
ccffnes'. These were simply learning outcomes which occurred in each of the 
five subject areas and did not claim to be complete or comprehensive. They 
were the ability to. 



access and select information 



synthesise and interpret information^ 



12 The group comprised representatives from. Ro\'er Group, Tioxide UK, WH Smith, GEC 
Alsthom, DEC Limited and the Association of Graduate Recruiters. 
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demonstrate self discipline (e.g. in managing time) 
manage personal stress 
communicate clearly and accurately 
work co-operatively 
work alone 



accept criticism 
I understand own strengths and weaknesses 
act ethically 

j possess basic computer skills. 

An edited list of the outcomes from each subject area was also included. The 
intention was to simplify the document and make it more accessible to a non 
academic readership. Despite this many correspondents found the document 
over complicated. 

"We all believe that the project is commendable ^ but the questionnaire 
conies across as academic overkill" 

7 found some of them incomprehensible" 

'The learning outcomes are described in many cases in terms that I 
find hard to relate to the day to day reality of graduate recruitment 
selection and employment. In a few cases I am even unclear as to the 
meaning of the tangle of words, " 

"Overall I feel that the tone of the document is rather academic." 



3.2.3 Personnel and line managers 

It also became clear during the workshop that graduate recruitment was often 
carried out by personnel staff, but that the learning outcome disciplines might 
be much more useful to a line manager. The workshop group also felt that it 
was the line managers of young graduates who could best contribute to the 
development of learning outcomes, since they had direct experience of the 
skills, qualities and specific knowledge which were required. There were how- 
ever likely to be significant problems in ccMitacting such people within organi- 
sations. This fear proved to be well grounded, and although letters and 
acccmpanying materials cleariy requested responses from Une managers, the 
majority of replies came from personnel sections, or from those with specific 
responsibility for graduate recruitment. Even where the names of line man- 
agers, or operational section heads were obtained, few responses were received. 
For example in one company only 2 replies were received to 64 copies of the 
document circulated to named staff. 
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3*2A Responses 

The majority of the 75 responses received were very general in nature, there 
were very few detailed reponses to the specific learning outcomes. The follow- 
ing are some examples. 

'The first point is a simple ambiguity where the questionnaire 
asks what one * expects*. There is a wide gap between what 
one expects and what one would like in most cases. I expect 
many graduates to be isolated from the real world of 
economic enterprise, naive but arrogant. None of these of 
course can be described as desirable qualities," 

"The authors admit that they have based it on academics* 
thoughts on desired learning outcomes. Perhaps that should 
have approached it rather from the employer s perspective, 
whereby the learning outcomes might have been usefully 
expanded to encompass more generic behaviours which are 
desirable within the business environment as opposed to that 
of academia." 

Some also misunderstood the projects intention to reflea the full range of out- 
comes of degree courses, and the values and outcomes of the different players. 

"I am certain that it would be reasonable to expect a 
graduate of English, to defamiliarise conventions, shared 
codes, meanings and values. Why that should interest us as 
employers is unclear, and is indicative of this study* s 
academic origins" 

The consultation with employers amply confirmed the fact that most of the 
employers responding were not looking for subject specialist knowledge. 

"As recruiters of graduates for 70 years I believe we are 
seeking two things in a graduate. Firstly, but only in some 
instances, specific skills and knowledge related to, for 
example, science and engineering. These learning outcomes 
would be too detailed and too dynamic to be specified. 
Secondly a package of personal skills and attitudes including, 
questioning, listening, analysing, thinking creatively, 
influencing, persuading, enthusing and potentially leading, we 
would seek these in graduates of any discipline." 

Even those who did stress subject knowledge also stressed the broader generic 
qualitites. 

"Except in the case of arts/social science graduates the first 
thing we are looking for is a considerable amount of detailed 
technical and professional knowledge about the specific 

subject they have studied this needs to be combined with 

the ability to harness this knowledge of facts and principles 
and systems to the solution of real problems. 

The second quality we are looking for is the capacity to 
assimilate substantial amounts of information, ideas, etc. 
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synthesise these and apply this knowledge to the resolution of 
practical problems. 

The third quality we look for is a capacity to work on the 
graduates own, without detailed supervision, to be a self 
starter. 

The fourth quality is a capacity to fit in and work as an 
effective member of a team, however we would also like 
qualities of leadership 

The fifth quality is communication skill 

Finally we would like to add commercial awareness which 
enables graduates to understand the realities of business 
problems and the relationship of specific tasks to overall 
economic objectives. The last is comparatively rare among 
engineering or social science/arts graduates. Generally 
speaking those who have developed it have not done so 
through their time it university or polytechnic. Business 
studies students are often aware of this side of life, 
unfortunately they are usually technologically illiterate so that 
it is hard to employ them in key roles in an engineering based 
industry, 

I am interested in your social science outcomes since these 
are our "graduates most likely to fail" category, they have 
excellent presentation skills, but generally do not stick to fact 
and have a "head in clouds" attitude, " 



3.2.5 General Learning Outcomes 

Several employers made suggestions about a general graduate list of outcomes. 
These included: 

ability to communicate with people at all levels 
numerate 

ability to plan and organise 
flexible 

show common sense 

sociable 

self motivated 

enthusiastic 

committed 
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work as a member of a team 



work on several tasks at once 



supervise and motivate others 



political and cultural sensitivity 



ability to develop holistic and visionary strategies. 

There were also suggestions that this list needed to be "much shorter", and 
contain 

"Some justification for both students arui employers as why 
these outcomes are relevant. Some of them are too specific 
and academic and not commutable to other situations arui as 
marketable skills" 

3.2.6 Comments from Design employers 

The circulation of consultative materials to graduate employers had concen- 
trated on large companies which recruited significant numbers of graduates. 
Design graduates were, however, more likely to find employment with small 
companies or to be self employed. A separate circulation of the materials to 
design employers was therefore undertaken, and 36 replies were received. 
There was a high degree of consensus about the key outcomes which they 
sought in graduates, with more than 30 of the respondents endorsing 8 of the 
statements: 



listen and observe 



be able to explain^ justify and present design solutions 



to influence others, eg to sell one*s own ideas and designs 



understand clients* problems and issues, and to help clients to 
understand problems and issues 



to accept and learn from failure 



gather and select appropriate information, synthesise information 
from different subject areas, and to understand information and 
relate it to the design opportunity 



continually add to their existing knowledge 



identify the major issues or problems in any design opportunity 



Learning Outcomes; student and employer views 



Very few of the identified outcomes were not thought relevant, and only 7 of 
the 36 respondents identified any in this category. 

The written comments were overall very positive. 

'1 found all of your questions basically deserved a positive 
reaction - all the points made on the questionnaire are 
basically the right ones. " 

However, there was a notable concern to stress the creative dimension of the 
study of design, and the less instrumental dimensions of degree level work: 

wlxat your questionnaire ignores on the whole is the 
importance of encouraging students to investigate and 
celebrate through their work personal ideas..,. Tools, 
techniques, applications etc. etc. can only be regarded as the 
"craft" required to enable the "art" to ... be realised...,!* ve 
seen far too many students wasting their time and potential by 
producing work that is expected of them rather than work that 
they want to do. ... Students should be encouraged to 
experiment, ... only through experiment and/ or failure will 
students actually acquire the mental ability to recognise the 
potential new idea." 

"Vm afraid I found the questionnaire rather dry and full of 
"art school speak". I think the major reasons that many 
people seek to work in design and the arts are those of love, 
excitement and enjoyment. Not to "select appropriate 

information" If I .. employ a graduate I would be happier 

if they knew that Giollo was an Italian painter rather than a 
make of pencil. " 

3.2.7 Overall findings on employer outcomes 

The employer consultations suggest that more attention should be paid to a set 
of generic learning outcomes which apply to all graduates regardless of subject. 
Their emphasis on such outcomes matches the student expectations of their 
degree courses, and suggests that students are conscious of some of the require- 
ments of the labour market. 

The lack of detailed comment on the subject specific learning outcomes was 
disappointing since these outcomes included many which are common to sev- 
eral different subjects, as well as outcomes which were thought to have a wider 
relevance to the worid of work. 

The contrast betwen the design and the other employers is notable, and might 
merit further examination to explore how far the difference relates to the size 
of organisation or the nature of the subject, and if the latter predominates, how 
far these are characteristics shared by other subject areas in the arts and be- 
yond. 
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3.3 Conclusions: students and employers 

All three groups of people involved in the project provided views on learning 
outcomes but, since the project concentrated on work with academic staff, it is 
not surprising that this yielded the most detailed information. 

Consultation with employers was on a limited scale and exploratory. It did not 
seek to produce systematic statistical data. It sought comment on the idea of 
describing the degree in terms of outcomes, as well as on the specific ccMitent. 
It would be naive to expect that employers would describe a single and easily 
assimilable set of outcomes, and in the event the range of responses was not 
unexpected. Overall it suggests that employers judge the employability of grad- 
uates through a small number of generic skills, and that, with the exception of 
the very specific skills sought in some roles (science and engineering) these 
were ccxnmon to most employers. The responses supported the learning out- 
come approach but, for employers, further work on the definition of broad 
generic graduate outcomes may be more valuable than the subject specific or 
vocational ones. 

The student workshops showed a similar trend. Students welcomed the oppor- 
tunity to discuss the outcomes of their courses, but tended to stress the personal 
development, social and vocational outcomes. 
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Chapter 4 



Assessing Learning 
Outcomes 

This chapter reviews the fmdings of the project on the 
assessment of learning outcomes. It draws on examples of 
assessment methods used and developed during the project 

4.1 Introduction 

The inclusion of the assessment of learning outcomes in the project design was 
intended simply to provide some illustrative examples of how assessment crite- 
ria could be developed. In the event however, assessment proved to be a much 
more important issue. There were several reasons for this: 

• the work required the groups to focus on the assessment of the outcomes 
they had described, this required both a hard look at existing practice and 
the development of new approaches to assessment. The validity of 
assessment (that is the extent to which the method(s) used actually 
measured what the staff were looking for) therefore became a primary 
ccxicem, whereas it was evident that previously the staff had been more 
concerned about the reliability (the extent to which the methods they used 
were consistent and reproducible). 

• many staff were concerned about the methods they were using to assess 
their students and about the difficulties they faced with increased student 
numbers. Some of the traditional approaches to assessment were 
impractical with large student numbers, and some staff faced an almost 
impossible burden of maricing, 

• there was general concern about, and interest in, developing new 
approaches to assessment but little expertise and experience, 

• there was confusion about recent develqpments, particularly about the 
assessment of competence and profiling and records of achievement, and 
the implications these had for grading and the classification of degrees. 

The woric on assessment was on a relatively small scale and this chapter there- 
fore presents a brief review of the methods used. 



The Project Prcx:ess 



The original UDACE model for the development of learning outcomes had 
described three approaches to assessment: criterion referenced, norm referenced 
and personally referenced: and had envisaged that all three approaches would 
have a place in the assessment of learning outcomes but would be of differing 
degrees of relevance to teachers, learners and employers. 

4.1.1 Starting points 

The work began with a two day residential workshop to review the current 
assessment practice of the participating academic staff and to plan the project 
work. Individuals were asked to describe what they assessed and how, and at 
what stage assessment was carried out in their courses. This was a helpful 
starting point. It provided an opportunity to unravel the complexity of assess- 
ment practices in some courses, and to debate and compare different ap- 
proaches. In a subsequent evaluation of the assessment work many participants 
described it as "one of the most useful things we have done on this project", 
and "the first time in my higher education career that I have ever had to justify 
what I was trying to do in assessment". 

Each of the subject based groups of staff was then asked to look at ways of 
assessing the learning outcomes they had described and to focus on one, or a 
small number of outcomes and trial fonms of assessment with their students. 
Most groups opted to assess learning outcomes which they felt were not as- 
sessed currentiy or were not assessed well. They were provided with a wide 
range of examples and ideas and with access to expert advice. 

Most of the discussion in the groups focused on the development of assessment 
criteria for the outcomes, notably on how these were to be decided and who 
was to be involved in that decision. 

The principle of using assessment fonmatively to provide feedback to students 
predominated in the plans for two reasons. Firstly this seemed neccessary if 
some of the more complex and personal outcomes were to be addressed, and 
secondly because of a very proper caution amongst staff about introducing 
radical changes of method without wider staff and student support, as well as 
support from external examiners. 

Each group undertook pilot assessment work. The following section outlines 
the work of three groups, in Design, English and Social Science. 



4.2 Assessment work in Design 

The Design group had a particular concern, to shift the emphasis in assessment 
from the product to the student. Their concern was that learning undertaken 
while producing the product is probably greater than the learning which can be 
observed in the product which is a major focus of assessment in Design educa- 
tion. The group was interested in using learning outcomes to develop better 
forms of feedback to students, and opted to use students themselves both to 



13 Understanding Learning Outcomes UDACE (1989) 

14 Nots 7 describes these three approaches in more detail 
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develop and agree the criteria for assessment and to assess one another in a 
peer group session. 

Teaching and assessment in Design does differ quite markedly from other 
courses, since the student's work is constantly on display and therefore open to 
comment and criticism from staff and students at all stages. In other subjects 
by contrast, a student may have several private attempts at a piece of work 
before s/he is required to present it for assessment, and then it may be assessed 
and commented on by a single member of staff. This could be the basis of an 
important distinctive learning outcome in Design, distinguishing it from other 
subjects where closed reflective processes are more common. 

18 second year graphic design students were asked to present a series of de- 
signs for a magazine relaunch. They began by using some of the general learn- 
ing outcomes developed during the project, for example: 

Analysis - demonstrate the ability to analyse information and images 
in order to communicate their meaning precisely, clearly and 
sensitively. 

These were discussed in small groups, v^dth staff playing a facilitative role, and 
specific criteria were agreed by the group. Each student was then required to 
present her/his designs and these were discussed using the agreed criteria by 
small groups of students. Each student was provided with written feedback 
from the rest of the group. 

The approach proved to be popular and successful, probably because it was 
well presented to the students, they saw it as beneficial to them and they could 
play a direct role in agreeing the criteria. The staff member concerned found 
that it provided clearer evidence of the learning outcomes than he was accus- 
tomed to obtain from such exercises. 

A similar approach was used with first year students in another Design course 
where the learning outcomes were presented as assessment criteria for project 
work. Again the staff felt that the exercise produced useful evidence of the 
learning outcomes, but the students resented the loss of a normative assessment 
grade provided by staff. For the students this respresented an abrupt change 
from a familiar form of assessment to an unfamiliar one. They did not have an 
opportunity to discuss the criteria which they felt were being imposed from the 
outside. They were clearly used to gauging their progress against one another 
and felt that the staff were there to provide assessment: "it's their job not ours " 
was a familiar comment. It appeared that discussion of the criteria for assess- 
ment was an important prerequisite for student involvement. 

4.3 Assessment work in E^nglish 

The English group developed an ambitious series of plans to assess * critical 
thinking' which they had described as an "overarching leaming outcome". 
They chose to do this because they felt that the term *cridcal thinking* was 
ambiguous and often not understood in the same way by staff and students; and 
that current methods to assess critical thinking, tended to focus on particular 
aspects of it, particulariy interpretation and analysis, but neglected the need to 
develop an independent perspective, or to identify unstated propositions or 
limitations in ideas. 
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In one institution, the learning outcomes were used to support a seminar pro- 
gramme, (which was accompanied by a programme of lectures), for two groups 
(16 in total) of students studying Eighteenth Century English Literature. Stu- 
dents were provided with two sets of documents: 

a seminar worksheet, describing the work the student was 
required to complete in preparation for the following weeks 
seminar. These were *text based', they listed the reading the 
student was required to complete (this was normal practice 
prior to the project) and detailed several of the learning 
outcomes (this was novel). The same learning outcomes did 
not appear in each seminar, although critical thinking was 
cleaily evident in most of them. 

a seminar ^attachment sheet', with three headings, critical 
thinking, research and information gathering, and creative 
thinking. Each was explained further with a short descriptive 
sentence. Students were asked to fill in against each heading, 
their comments on their own preparation for the seminar, these 
were briefly discussed at the start of each class. At the end of 
the seminar they were asked to fill in comments on the 
seminar outcomes under the same headings. 

The staff member kept a record of all the student sheets and carried out an 
evaluation with students at the end of the series of seminars. 

The exercise was judged to be successful by both the member of staff and the 
students. The lecturer and the students made the following comments; 

• "the course as a y^hole seemed "more sharply defined", 

• "student planning for the seminars was more directed 
and more precise " 

• "students returned to the topics more frequently during 
the week preceding the seminar," 

• "differences in student responses became more marked as 
the seminars progressed, (it emphasised or 'stretched 
out' differences in student performance) " 

• "the seminar 'had greater authority*, and was taken 
more seriously by students, even though the outcomes did 
not contribute directly (arithmetically) to the degree" 

• "students felt they were forced' into being more self 
reflective and self critical (in an academic sense), this 
was a novel experience for many of them," 

• "there was an increase in the awareness of the seminar 
as a group activity, rather then something which a 
collection of individuals came together to do" 

• "keeping a record of seminar preparation had a direct 
benefit on learning. " 



6i 



ERIC 44 



Assessing Learning Outcomes 



Eighteenth Century Literature 
Seminar Worksheets 

Week 5 

Pope and Fielding 

1 List parallels and differences in Pope's and Fielding's use of 'inflationVmock-heroic' 

2 Is Fielding interested in different levels of society from Pope? 

3 Imagine Pope's response in reading Fielding 

4 Consider the rcIaUonship between a) Pope and his reader b) Fielding and his reader. Are 
there notable differences? 

5 Imagine a 18ih ccntuiy Victorian moralist (family man concerned about what his teenage 
daughters should/should not read) bowdlcrising Tom Jones. What might he feel 
compelled to leave out? Would he have to apply the same treatment to Pope? 

Finally, attempt to summarise from your reading in weeks 1-5 the 18th century attitude to 
classic sundards and values. Does the classical world represent superior values, or is 
this a more complex issue? 



Week 9 

Swift - Gullivers Travels, Parts I and II 

1 Allocate each of the following extraas to their immediate context, explaining their satirical 
significance: 

a) Whoever performs his part with most agility, and holds oul the longest in leeping and 
creeping, is rewarded with the btue-coioured silk 

b) Truly, I had little to say in return, farther than the common answer, that different nations 
had different customs 

c) / suppose his faculty of smelling was as nice with regards to me, as mine was to that of 
his people.... 

d) As for himself he protested, that although few things delighted him so much as new 
discoveries in art and in nature; yet he would rather lose half his kingdom than be 
privy to such a secret. 

2 In Part I, make a list of those aspeas of Lilliput that Swift/Gulliver appears to regard as 
positive, and those regarded as negative. 

3 Do you find any progressive changes in Gulliver's attitudes and experience between PaiU I 
and II? If so, what exactly are those changes? 

4 Part I is frequently listed among books read by children. 

a) Why do you consider il interests children? Is it likely to be of uniform interest to younger 
readers? 

b) Is Part II suitable for children? If so, why and in what respeas? If not, why not? 



Figure 4.1 - English: an Approach to Assessment 
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Seminar attachment sheet 



Term: Student: 
Week: 



Worksheet Items 


Comments on Seminar 
Preparation 


Comments on Seminar 
Outcomes 


1 Research/Information 
Gathering 

(How muCh/how well 
prepared? Use of library 
resources? Task sufficiently 
well specified? fSfficuIties 
encountered?) 






2 Critical thinking 
(How much 

analysis/synthesis involved? 
How much 

interpretation/di scussion 
required? Number/relevance 
of examples cited?) 






3 Creative thinking 
(Usefulness/relevance? 
Limitations? Difficulties? 
Insight?) 







Seminar Summary 





outstanding 


good 


satisfactory, 
with some 
inadequacies 


inadequate 


Usefulness 










Relevance 










Contribution (of scIQ 










Contribution (of others) 











Go 
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Again the exercise demonstrated that stating the learning outcomes provided 
better evidence that students v^ere (or were nor achieving them. The benefits 
to the learning process were also very clear, the motivational advantages of this 
approach are described by Eraut (1989) writing on the advantages of sequenced 
objectives, 'Trom the student's point of view, what probably matters most is 
an objective's position on the immediacy-remoteness continuum,,,. Many ob- 
jectives can appear to be both temporally and conceptually „,remote. Per- 
ceiving links between their immediate objectives and possible ultimate goals 
can be crucial for some students motivation,'' 

4.4 Assessment work in Social Science 

The Social Science group chose to concentrate their attention on three learning 
outcomes which they felt were not currently assessed, 

open-mindedness in the face of unwelcome ideas and evidence. 



the ability to suspend personal judgments until appropriate evidence 
had been gathered, analysed and evaluated. 



competence at taking the role of the other, that is the ability to 
represent the views of another person without personal bias. 

The group felt that these were implicit in the teaching of many social science 
courses, but that students were probably not so clearly aware of what they 
meant in practice. 

One of the pilots was undertaken with a group of 68 second year students from 
a module on "Ethics and Social Sciences". The assessment of these particular 
learning outcomes required the design of a specific learning activity (a series of 
seminars and debates) to provide opportunities to understand, develop and 
assess these learning outcomes. The whole process had five stages: 

1. Students were given a list of seminar topics (they were also advised 
that they could suggest their own if they wished), and details of 

the assignment they had to complete. This was the only 
summatively assessed component of the course and contributed to 
the degree classification. 

They were also given a list of learning outcomes which they were 
expected to demonstrate (these included the three above) and 
which would form criteria for assessment of the assignment. 

2. The first seminar was used to explain and discuss the list of learning 
outcomes (developed in the project) provided by the tutor, this 
included an explanation of the differences between norm and 
criterion referenced assessment and the purposes of the UDACE 
project. Students were asked, in small groups, to develop criteria 
which would enable an assessor to judge whether or not a student 

had demonstrated the learning outcomes, 

3. The second seminar was used to stage a series of role plays which 
debated a range of ethical problems (the actual example used was 
of toxic chemical disposal, and the role plays focused on how a 
range of people reached decisions, the final stage being a 
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prosecution for reckless endangerment). Students took part in the 
role plays and acted as observers, they used the learning 
outcomes and criteria developed previously to inform their 
preparation, performance and observations. 

4. The third seminar was used to stage a debate using a case study (this 
time of a bank fraud), students were divided into groups and were 
asked to take either a Kantian or a Maclilavellian position in the 
debate. 

5. The last stage was the writing of the assignment, students were 
required to self assess their assignments against the learning 
outcomes and criteria discussed in the first seminar. The tutor 
used the same criteria for assessment and his comments were 
made on the same self assessment form completed by the students. 

Comments from both staff and students were used in evaluating the exercise. 
The overall findings indicated that, despite the discussions, some students were 
still unclear about some of the learning outcomes and assessment criteria: and 
this was reflected in the final assignments as well as in student comments. 

This suggested that although the member of staff concerned thought the out- 
come statements were clear, his students were able to reinterpret and misinter- 
pret them in various ways. For example: 

• students saw some of Uie outcomes as "all or nothing" statements, some 
of them felt that the outcomes were too difficult and complex and that 
they (the students) were some way from achieving them (cf Eraut op cit), 

• some students suggested that the learning outcomes were attempts to 
change^ rather than develop, their values and opinions whereas tlie tutor 
felt that they were intended to describe how students examined and 
understood values, and how they developed ways of reviewing and 
testing them through a process of critical scrutiny of evidence. 

• the tutor also commented that the ability of students to evaluate their own 
woiic was a major weakness, and that this probably needed to be a 
learning outcome in its own right. 

• students suggested that it was confusing for them use a learning outcomes 
approach in one part of the course. The tutor commented that the ideas 
needed to be reinforced, otherwise the benefits were dissipated, since 
students do "what is required of them" on the learning outcomes parts of 
the course and resume "normal" worldng elsewhere. "The whole course 
team needs to be committed to the approach and this implies the 
involvement of external examiners as well as the institutional quality 
assurance procedures". 



15 These issues are discussed at length by Schon (1987) 
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4.5 Conclusions; Assessment 

4.5.1 The initial workshop 

Tlie workshop established several points very cleariy: 

• there was little evidence of the development of an assessment strategy for 
a course or a programme of study. Individual staff often had litUe idea 
about what or how their colleagues were assessing 

• most approaches to assessment were very conservative: some staff had 
never thought of assessment in terms of anything but an essay or an 
unseen examination, and most of the assessment methods being used 
relied solely on tutor assessment, 

• most staff placed heavy emphasis on reliability and grading, although 
several had evidence of variability in the assessment of written woric 
(such evidence of course abounds in the literature). The ways of grading 
and marking work and the distinctions between grades and classes, 
appeared to occupy a disproportionate amount of time and effort, 

. assessment systems depended on sampling approaches whose validity 
seemed questionable, if the purpose is to estabUsh that the candidate has 
demonstrated achievement of a specified range of outcomes. An outcome 
led approach raises questions about assessment strategies based on 
achieving a limited percentage of marks on an examination which allows 
the candidate to select a few questions from a long list. 

4.5.2 The development of plans 

The work on assessment was disappointing in that it produced fewer illustrative 
examples of assessment practice than anticipated. The plans developed proved 
to be over ambitious in practice since several of the participating institutions 
faUed to complete or report on their work. There appeared to be several reasons 
for this: 

. shortage of time, some staff found it impossible to make time to introduce 
their plans, 

• difficulties in changing assessment style in mid course, without support 
from other staff. 

• lack of institutional support and co-operation from colleagues. 

These findings are important in considering how changes of this kind might 
actually be effected. 
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4,5.3 Assessment and the learning outcome 
statements 

nie most important finding of the work was that the requirement to assess the 
learning outcomes forced the subject groups to clarify their learning outcome 
statements, and made the groups realise the importance of accuracy and nego- 
tiation when defining learning )utcomes. 

This led some staff to feel that the learning outcomes approach was not as 
instmmental as they had feared, since it required staff and students to talk 
about what they were trying to do, but stiU pemiitted students some freedom of 
expression, and did not discourage other unplanned learning outcomes. 

The work also led some of the groups to think that there were learning out- 
comes which they had not described, but which should be included. The social 
science group for example, decided that they needed further learning outcome 
in self- reflection and self assessment. 

OveraU this suggests that the description and assessment of learning outcomes 
need to proceed hand in hand, and that the requirement to be able to assess 
what is being described is a useful discipline. 



4.5.4 Assessment criteria 

The evidence of diversity suggests that the description of learning outcomes 
should not be based on a set of nationaUy imposed performance criteria, as is 
the case with learning outcomes for NVQs. The group approaches to setting 
perfonnance criteria for assessment were interesting, the most successful in- 
volving discussion and agreement of criteria with students, rather than their 
ii iposition. This does not imply that students should be given a 'free hand' to 
decide criteria, but the discussions surrounding the setting of criteria plainly 
took place in a setting where students were expected to, and responded to the 
opportunity to, discuss and understand criteria for themselves. 

Discussion with some of the groups suggested that the ability of students to 
decide their own assessment criteria was in fact a learning outcome. This 
linked back to some of the eariier debates about the culture of disciplines, and 
the idea that a higher education course was a means of admission to that cul- 
ture. The principle was not so much that students should be able to assess 
themselves against simple criteria, as that they should have learned some of the 
"syntax and concepts" (Schwab 1962) of their subjects/disciplines and interna- 
lising the discipUne based value systems against which they were being as- 
sessed. 

For some groups there remained some 'learning gains', for example personal 
growth and development of the student, which academic staff felt were either 
impossible to describe adequately, or impossible to specify in advance. This 
was a source of considerable concern at the outset of the project, but the dis- 
cussion and clarification which surrounded the assessment of learning out- 
comes, in particular the encouragement of unplanned outcomes, helped to aUay 
some of them. 

The findings suggest that one of the major differences between learning out- 
comes in NVQs, and learning outcomes in higher education, is the way in 
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which perfomnarice criteria are arrived at. NVQs are in a sense a 'licence to 
practice* and it is appropriate that the criteria for judgment should be nationally 
decided by the bodies concerned, and presented in some detail. It is not clear 
that higher education needs to work to nationally decided and pre-defined crite- 
ria in all areas, although therc are cleariy cases where degrees lead to profes- 
sional qualifications and external standards will need to be used. The ability to 
develop and use one's own criteria may be an important defining characteristic 
of higher education. 

4.5.5 Grading 

Several of the groups which chose to discuss and agree criteria with their stu- 
dents were subsequently faced with the question of how to grade tlie work. 
There was some discussion of the necessity of grading since the agreement of 
criteria was essentially a statement of what was required If the students met 
the criteria, was it really necessary to grade them? Grading was however stand- 
ard practice, and was the basis of the classification of degrees, its abandonment 
posed questions of a serious and fundamental nature, and the project did not 
provide sufficient evidence to propose a change of this nature. Nevertheless, 
the grading issue is a fundamental one in the reshaping of higher education, 
and will need to be addressed in any further development of an outcome based 
approach. 

4.3.6 VaUdity and ReliabiUty 

The validity and reliability of assessment were repeatedly discussed in connec- 
tion with this work, although the difference between the two was often not well 
understood. There was general concern with the reliability of the forms of as- 
sessment used, this mainly centred on maricer reliability and second marking, 
but it was not until the groups came to consider how they were going to assess 
the learning outcomes they had described, that they really thought seriously 
about whether the methods they were using assessed the outcomes they were 
looking for. 

The requirement to find ways of assessing the learning outcome statements 
prompted a more serious evaluation of the validity of the proposed assessment 
methods than was expected. This in itself may make woric on learning out- 
comes valuable to many institutions. 

The discussion prompted some groups to ask what their existing approaches to 
assessment aaually measured. It was not clear that written examinations as- 
sessed some of the learning outcomes, and it was also not clear that course- 
woric essays were any better at assessing some of them. The current system of 
grading, and using a 40% pass marie also permitted some students to pro- 
ceed/pass without requiring them to demonstrate that they had achieved all the 
outcomes. Current practice might allow a students failure in one area of a 
course (and hence presumably in the learning outcomes associatea with it), to 
be compensated for by good performance in another. For some learning out- 
comes this was probably not contentious, however it did raise a serious ques- 
tion about whether some of the learning outcomes should be essential 
requirements, which students should be required to demonstrate before they 
passed. 
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The findings strengthened the view that there were learning outcomes which 
were core, or essential in particular subjects, and that their definition had im- 
portant implications for assessment practice. 

4.5.7 Over-assessment 

The groups had expressed concern tliat the deiscription of learning outcomes 
would result in an increase in assessment, with the danger that students would 
be * over- assessed'. This was also suggested in some of the responses to the 
consultative document. Actual experience suggested that the nature of assess- 
ment in higher education was complex, and that it was impossible to assess one 
learning outcome without also providing evidence of others. 

This made it clear that learning outcomes at this level need not be assessed one 
by one: a single assessment process might provide e\ddence of several different 
learning outcomes. It might thei"efore be helpful both to provide a student with 
several opportunities to demonstrate a particular learning outcome, and even to 
require that they were able to demonstrate it more than once, in different ways. 

It also suggested that once staff were alerted to the idea of looking for learning 
outcomes, then they might discover that a student had demonstrated other 
learning outcomes, not described and anticipated. This supported the view that 
the concept of learning outcomes was not as instrumental as had been feared. 

4.5.8 Benefits for staff 

The woric cm assessment was thought by those involved to be a useful experi- 
ential learning exercise which provided valuable information on writing and 
implementing learning outcomes, as well as on assessing them. The exercise 
was simple to organise and to implement, and cross institutional work of this 
kind may be a model which could be used elsewhere. Some of the findings and 
conclusions reached are: 

• the projea allowed staff the opportunity to stand back and consider what 
they were assessing and why? This provided a novel and much welcomed 
opportunity to question existing practice and received wisdom in 
assessment. 

• the work persuaded several tutors to become "classroom researchers" (in 
the sense described by Patricia Cross (1988) and to apply the critical and 
analytical skills which they commonly reserved for the study of their 
subjects, to the processes of teaching, learning and assessment. One of the 
respondents to the consultation document commented, " I have however 
come to recognize that academics tend to abandon their usual standards of 
argument and evidence entirely when discussing matters to do with 
teaching and learning". 

The need for staff development in assessment is evident, but the experience of 
this project suggests that such development can be effectively and economi- 
cally achieved through subject networks and * classroom research' rather then 
through training programmes. 



ERIC 52 



Assessing Learning Outcomes 



4«5*9 Learning outcomes, assessment and 
teaching quality 

The work on assessment prompted some groups to examine what and how they 
taught their students, and the extent to which the methods of teaching and 
learning thf^v used helped students to achieve the outcomes they had described. 
There wo^ an acknowledged tendency to assume that lectures, tutorials and 
seminars were learning environments, and that acquiring knowledge through 
these environments naturally led to acquiring knowledge processing skills. It 
was acknowledged that many of these skills were held to be implicit by staff 
and that some students never really understood what was required of them. It 
was clear that describing these implicit expectations as learning outcomes 
would be an advantage for students, but that describing them was only part of 
the aaswer, students needed to be provided with an opportunity to learn and 
demonstrate them as well. There was some doubt as to whether many courses 
did this well, and this posed a problem for staff and students, since the failure 
of a student to demonstrate a particular learning outcome might be a failure of 
the course to provide an opportunity to develop it, rather than a failure of the 
student to learn it. 

This is an interesting finding since it provides an example of staff using learn- 
ing outcomes as a means of examining the quality of their courses, rather than 
their students. Existing approaches to course validation and review use similar 
questions, but it may be that learning outcome statements together witli details 
of their assessment could provide information which helpfully contributes to 
such processes. 

4.5.10 Implementation: work with students 

The work shows that it was not easy to implement a learning outcomes ap- 
proach on one part of a course in isolation. It was only possible to do some of 
this work because students were prepared to co-operate. The fact that all the 
project participants did not complete their work on assessment suggests that 
co-operation may have been a difficulty in some institutions. 

The project did cause some problems for students, some had difficulty in ad- 
justing from one form of assessment to another and others resented the require- 
ment to use and discuss criteria for assessment. :Dreferring a familiar system of 
grading administered by the tutor. 

Overall it was clear that a learning outcomes approach could not be developed 
piecemeal, it required the collaboration of the whole course team, and this im- 
plied also the involvement of external examiners and course quality mechan- 
isms. 

The project predictably raised many questions about embedding innovation and 
the management of change in higher education. It underlined the need to sup- 
port innovation by individuals, but to avoid the Mone hero/ine syndrome* en- 
countered so frequently. Support is probably best provided through existing 
course structures and quality assurance procedures. 
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The development of a learning outcomes approach would clearly have conse- 
quences for current approaches to assessment, a number of specific issues can 
be identified: 

• if learning outcomes were to be a clear and public statement of what a 
student was required to be able to do, then there would have to be clear 
decision on what constituted satisfactory achievement for each learning 
outcome. 

• there would also have to be a clear decision on whether a student should 
be required to achieve a "satisfactory performance in each outcome, or in 
some outcomes, or should there be a predetermined minimum number of 
outcomes (drawn from each type?) which a student must complete 

, satisfactorily to proceed, or to achieve an award. 

• there would need to be clear guidelines on whether it was possible or 
desirable to grade all assessments, would it be more practical to grade the 
assessment of some learning outcomes, have others as pass and fail? 
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Accreditation: major issues 



The ivork on accreditation raised three mc^or issues of 
principle. These concerned the notions of level credit value, 
and coherence, and are discussed below. Chapter 6 then offers 
examples of approaches adopted by three of the subject groups. 

5.1 Introduction 

An investigation of the relationship between learning outcomes and credits was 
a major objective of the project. The principles of credit accumulation were 
well established in higher education through the CNAA CAT (credit accumula- 
tion and transfer) scheme. Standards within the scheme (at undergraduate level) 
are determined with reference to the principal award, the Honours degree, and 
the unit of credit is not defined. 

The major disadvantage of this approach is that it assumes that all degrees, 
irrespective of subject or awarding institution, are of equal value. In practice 
the unit of credit acts, overtly or covertly, as the standard and is measured in 
fractions of the time it takes to complete a degree course, rather than as a set of 
achievements or standards. This approach has further implications for the ac- 
creditation of any learning which takes place outside a course structure, all of 
which must be measured against the taught course benchmark. An outcome led 
approach might help to clarify what credit was being awarded for, and provide 
a more robust basis for credit transfer. 

This is particularly important in view of developing interest in the creation of 
some form of national CAT framework, embracing all post-school learning, 
and enabling progression from basic education to postgraduate work. This 
would call for clearer relationships between the three major credit accumula- 
tion and transfer systems (National Vocational Qualifications, CNAA CAT, 
and the developing National Open College Netwoik^^. Explicit outcomes 
seemed likely to provide a better basis for such a development than the use of 
learning time, when notions of time are very variable across sectors and institu- 
tions. 

The project therefore set out to investigate approaches to determining credit 
value based on learning outcomes, rather than actual or notional time. 



1 6 These three schemes are outlined in Note 5 
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5.1.1 Credits and delivery systems 

It is important to stress the distinction between credits, which are measures of 
learning and achievement, and units or modules, which are delivery systems for 
courses. Many degree courses are offered to students through a system of mo- 
dules and units, which are simply blocks of teaching, A unit/module might for 
example, occupy one tenth of student's time in an academic year, and a student 
might need to complete ten units in order to fulfil the requirements at that 
level. 

For many full time students on convention courses the notion of credit is 
irrelevant, since the programme is based on the completion of set number of 
taught blocks. However a student (full or part time) might transfer to another 
institution, or choose to take part of his/her course at a second institution. Here 
there might be a different delivery system perhaps with much larger units occu- 
pying one quarter of a student's year. The common credit structure (120 credits 
at each of three levels) provides a useful framework against which a variety of 
delivery systems can be compared. A student might also receive credit for prior 
learning or for v/ork based learning, and hen again the credit structure pro- 
vides a useful and simple means of calculating the relative worth of the learn- 
ing, in comparison with a degree, 

5.1.2 Starting points 

The guiding principle of CATS is that it provides the means of accrediting 
learning, wherever the learning took place, provided that it can be assessed. 
Each of the five subject groups was asked to develop ways of determining a 
credit value for their learning outcomes using the CNAA CAT scheme. 

The work c«i credits and learning outcomes was the most complicated, and in 
some senses the least successful area of work. Most of the problems stemmed 
from a lack of understanding of tlie CAT credit scheme, and its implications 
for the accreditation of learning. Most of the group members had little experi- 
ence of CNAA CATS, or of credit accumulation and transfer generally, and it 
was therefore necessary to provide a workshop and materials for participating 
staff. This was successful in transmitting the basic information but as later 
work showed, the groups really needed more in depth experience. 

Three major issues were raised by the groups in the subsequent work, these are 
questions of level, credit value and coherence, and are discussed below. 
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5.2 Major Issues - Levels 

Course diversity 

It was agreed that, initially, subject groups would not examine levels within 
degree programmes, to avoid discussion of the outcomes of the degree becom- 
ing confused with debates about particular course stmctures. The wisdom of 
this decision became evident as the variety of course structures, even within the 
same subject area, was identified, with material included in the first year of 
some courses occurring in the second or final year of others. This reflected the 
very different ethos of some courses and different approaches tc course design. 
The following examples illustrate the degree of diversity: 

Courses where the first year was devoted to fundamental 
principles or areas of the subject, which were then built on in 
the second year, with the third year covering applications of 
this information in different settings. 

Courses which began with a broad consideration of 
applications and what one member called "real world 
problems", and then developed specific knowledge relating to 
these in the remaining years of the course. 

Courses with a common second and third year, offering the 
same options to students from both years. 

Courses with a very structured first two years and a 
concentration on individual project work in the final year. 

Courses were made up of units where the level was not 
defined, a student could therefore take units at any stage in 
his/her programme. 

At first it was felt levels might be used to describe stages in a course, rather 
than as absolute descriptions of levels of attainment. The O^AA CATS Regu- 
lations (CNAA 1991) were found confusing on this issue. They first propose 
that levels relate to the standard of work, and not necessarily to the year, or 
stage of the course at which the work (module, unit) is normally encountered. 
They then go on to relate standards to the year of the course in which the work 
occurs. On this basis learning at level 2 for one student could be coimted at 
level 3 for another. This is helpful in the construction of progranmies of study 
for CAT suidents since it allows greater flexibility. A student studying at Level 
3, who wished to broaden her programme of study could therefore use an 
agreed proportion of her programme to include a modem language, or finance 
units taken from Level 1 or 2 of other courses. In this case it is the develop- 
ment of the student, not the unit itself, which determines the level. This fur- 
ther implies that levels are stages of progression, intended to provide a basis 
for the construction of a coherent course for an individual studeiu. 
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5.2.2 Two views of level 

Overall there were two broad approaches to the question of level, which may 
coexist within the same degree programme. The first was based on the notion 
of progression or prerequisites, and the Social Science and the Design groups 
both produced models based on this principle. Other groups suggested that the 
same learning outcome statements could appear at several levels, usually by 
making the criteria for assessment more demanding at each. The second ap- 
proach rested on the notion of a continuing developmental process through the 
degree program ^ne and sought a central developmental strand against which to 
describe this progression. The clearest version of this was offered by one in- 
stitution in response to the consultative document 

Incremental Progression 

On this view learning outcomes statements should describe what will be 
achieved at the point of graduation, but it might be necessary to demonstrate 
some of these at an earlier stage, depending on the construction of the course, 
Some learning outcomes might therefore act as requirements for progression to 
higher levels within the programme, and some could not be demonstrated at 
earlier stages. Table 5.1 shows how an example of how the Social Science 
learning outcomes were allocated to an existing course structure. The first row 
in the table lists the learning outcomes which the student must have satisfactor- 
ily demonstrated at the start of the second year of the course. 



Social Science: Learning Outcomes Allocated by course & year: Year n 


LOs dcfiniiivcly achieved by suit of year 2 




LOll, 13a. 13b. 13c, 16. 22 


GenenI LOi across all year 2 couises 




LOl, 7. 9, 10. 14. 17. 18. 19, 20. 24. 25, 26 


Sociology of Culture 


20 


L02. 3. 4 


Theories of Modem Society 


20 


L02. 3. 4 


Methods of Social Inquiry II 


20 


LOS, 6. 8. 12. 13d. 13c, 21 


Year 11 Options (3 to be chosen) 






Inequalities and the Suie 


20 


L03 


Inequalitites: Reproduction or Change? 


20 


L02.3. 5, 12.15 


Crime, Deviance and Society 


20 


L02. 3. 4, 5 


Systems of Justice 


20 


L02.3. 12 


Woric. Employment and Society 


20 


L02. 3. 26. 27 


rr and Society 


20 


L03. 12. 13d. 13c 


Language 11 


20 


L013f 


Ethics and Social Philosophy 


20 


L04. 12, j5. 21. 23. 27 Subtoul=l20 


Index of Learning Outcomes 






LOl: Vocabulary 


Ij02: Socio- historical Development 


L03 : Major Social Institutions 


Ij04: Theoretical Perspectives 


LOS: Forms of ResearchA^alidity of Results 


L06: Conceptual Frameworks for Research 


L07: Access, Sort, Interpret Information 


LOS: Practical Research Experience 


Ij09: Contcsubiliiy of Social Judgement 


LOlO: Argumenution 


LOl 1: Suspension of Personal Bias 


LOl 2: Ethics in Social Science 


LOl 3: Communication Skills i • Liicrmcy b - Ntne-ukinf c 


- Keportage d • Numeracy c - Computing f - Forcigin Language 


1014: Self-direction 


L015: Self-knowledge 


LOl 6: Deadlines and Pressure 


L017: Critical Attitude 


LOl 8: Organisation 


L019: Synihcsise and Report 


IjO20: Know teaching methods 


L021: 


Take role of the other 


L022! Reaction to Feedback 


L023: Ethical Conduct 


Iv024: Tcamworking/Teamlcading 


Ij025: Practical Appraisal of Group Dynamics 


IX)26: Reform and Design of Groups 


IX)27: Professional Intervention 



Figure 5.1 - Social Science: Levels and Progression 
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Continuous Development 

The alternative model also required that learning outcomes should be described 
at the graduate level but proposed that level should be defined, not in terms of 
prerequisites but on the notion of continuing personal development. 

One cation proposed suggested that the best basis for such criteria would be 
the increasing autonomy of the student. The eaily stages of a course were 
usually largely defined and delivered by the staff. Information was therefore 
prepared for the student and the course covered the basics or fundamentals of 
the subjea. At higher levels the student was increasingly expected to retrieve, 
select and process information independentiy and to develop a wider range of 
learning and problem solving skills. At the final level the student was expected 
to think and learn independentiy, develop self motivation and demonstrate in- 
tellectual curiosity in developing and exploring novel ideas and solutions. In 
tiiis case, die subject material might be identical in the first and tiiird years of 
the degree course: what changed was the degree of sophistication with wliich 
the indvidual learner was processing this. 

Figure 5.2 shows a model of levels based on learner autonomy, which was 
offered in the course of Uie project consultation by Falmouth School of Art and 
Design, where it is used as the basis for assessment in a degree in Graphic 
Information Design. 



Falmouth School of Art and Design 
Assessment Levels 

Level I 

This is learning hat students perceive is done to theni by teachers, rather than something 
they do. Learning is memorising. The student has an active role in memorising, but the 
information being memorised is not Cransfonned in any way. 

Level 2 

This is learning that is acquiring facts, skills of procedures which are to be used. What is 
learned is seen as needed in order to do things at a later date, but there is still no 
transformation of what is learnt by the learner. 

Level 3 

This is learning that makes sense. The student makes active attempts to abstract meaning 
in the process of learning. 

Level 4 

This is learning as understanding reality. Learning enables the student to perceive the 
world differently. The student sees her work in a world context and recognises its value. 



Figure 52 - Falmouth: Levels and Autonomy 
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5.3 Major issues - credit rating 

The second major issue concerned credit rating - allocating a credit value to an 
outcome or set of outcomes. This is the measure of Volume' in a CAT scheme 
and is problematic in relation to outcomes, which are inevitably of different 
sizes. Even when clustered into units they are unlikely to be of equivalent 
value, as the NVQ system, with units of very different sizes demonstrates. The 
CNAA CAT scheme aims to provide maximum flexibility and to accommodate 
the structure, objectives and learning outcomes of many different degree pro- 
grammes and learning at degree level. It is based m the notion of the Honours 
Degree, which is then divided into a number of smaller 'pieces'. The CNAA 
divides the degree into 360 equal parts (^credits'), although it suggests that 4 
credits is the smallest useful unit for teaching or assessment . 

The major weakness of the scheme is that it relies on the degree as a bench- 
mark, assuming an equivalence between degrees from different institutions and 
between degrees in different subjects in the same institution. 

The CNAA CAT scheme does not define what a credit represents other than by 
reference to a full degree, and a credit has value only in relationship to the 
whole (the degree). Intermediate awards, such as Certificate of Higher Educa- 
tion (120 credits) and Diploma of Higher Education (240 credits) are recog- 
nised as stages or milestones on the way to a degree, although like all higher 
education awards they are expected to represent a coherent body of learning, 
and not a random collection of credits. The definition and award of credit is a 
matter for the quality assurance procedures of the individual institution. 

The working definition of a credit is achieved in two major ways which are 
described below. 

5*3*1 Credits related to courses 

The credit rating of any piece of learning is usually determined with reference 
to an existing degree course suiicture. For example, a modular degree coui*se 
might require a student to complete 5 modules per term, and a total oi fifteen 
modules per year. Using CAT credits, each module would carry 8 credit points 
and a student would accumulate 120 credits for successful completion of the 
fifteen modules. 

An alternative approach, adopted for example, by Liverpool Polytechnic, 
begins by defining a credit in terms of time. The scheme there begins by defin- 
ing a credit as "one hour of structured, directed learning activity undertaken by 
a student per week for a 30 week learning year" (i.e. including teaching, woric- 
shop and laboratory time and independent study time, without which enormous 
discrepancies appear between courses in different subjects). So a credit is 
defined as 30 "notional hours". It then defines "the minimum learning require- 
ment for any student" as 900 hours per year: a student therefore accumulates 
30 credits per year and 90 credits for an Honours degree. One Liverpool credit 
is equal to 4 CAT credits, the smallest unit recommended by the CNAA. 



17 CNAA Handbook (1991). 
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5.3.2 Credits not related to courses 

The previous options above use modular structures or notional time to create a 
benchmark, and courses, units and modules are therefore credit rated using 
time. Learning which is not acquired as a result of a course (for example prior 
learning, or woric based learning) is usually compared to ccMiventional courses, 
and learning times and course objectives matched, to obtain a credit rating. 

APL, in-company learning and work-based learning can pose problems if 
credits are calculated using time. The comparison may be difficult because of 
the large mismatch between actual time and credit value. An individual may, 
for example, claim credits for experienrial learning, through APL or a learning 
contract. The learning outcomes may be the same as those of the course, but 
since they have been gained experienrially they may have taken much longer to 
acquire. On the other hand the same learning may be part of a training course 
mn by a large employing organisation, where the same material may have been 
covered in a much shorter period of time than on a degree course. 

More complex problems are posed, however, by learning which is not part of 
an existing course, and where no direct comparison of this kind can be made. 
Some prior learning falls into this category, and here estimates of level and 
credit number are made by careful examination of the assessment criteria, 
which give an indication of the level, and by looking at the learning in relation 
to the whole award. Here questions of balance, coherence and content are used 
to ask questions like, how much of the award does this learning represent?, is 
the learning at the same depth as that on existing degree courses?, is there 
sufficient breadth of learning in comparison with other degree courses? 

The drawback of all these approaches is that they use the three year de ;ee 
course as a benchmark, and do not define what the degree requires. As one 
employer correspondent said: 

'Why are degrees three year courses? - is it because the 
government funds you for three years, I expect that if you 
were only funded for two years that some institutions would 
fairly quickly develop a two year course". 

Another suggestion was that degrees needed to take three years because they 
took into account a maturation factor for their traditional 18-21 year old clien- 
tele. Several people suggested that degrees could take two years if the age at 
entry was 21 not 18. Current work on development of accelerated degree pro- 
grammes may address some of these questions. 
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5.4 Major issues: coherence 

The third issue concerned coherence, it was clear from the CATS regulations, 
and indeed from the CNAA Principles that credit accumulation was not simply 
a matter of numbers. A student would not get a degree just by accumulating a 
random collection of 360 credits. An award must embody some notion of co- 
herence and balance to be met in the learning which the credits represented. 
This requirement imderpinned the differentiation between general and specific 
credits. The O^AA made clear that general credits were: 

"a numerical value representing the credit points attached to 
qualifiable prior learning of an intending student without this 
being considered in relation to a particular programme of 
study", (CNAA Handbook 1991 

Whereas specific credits were, 

"directly relevant to an individual student's particular 
programme of learning", 

A student might therefore gain general credits by the completion of one module 
on a course or by APL, but these would only become specific credits when 
accepted as part of an approved programme of study leading to an award. 

The question of coherence was a useful one to consider in looking at how 
learning outcomes related to credits. What did coherence mean? and who de~ 
cided what was coherent or not? 

In theory there is a range of possibilities, from apparently free student choice (a 
*pick and mix' approach) to a restricted number of routes, or pathways, 
through the scheme. A student was usually able to choose a number of themes, 
or major and minor subjects or major subjects with elective choices in different 
areas. Choice was often restricted to some options within a route, and there 
were other restrictions imposed by timetabling and prerequisites (one module 
had to be completed before or with another). The amount of flexibility varied 
from one institution to another, but it was clear that there were a range of rules, 
developed by academic staff, which governed coherence. 

Some institutions offered students a greater choice than others, and some had 
more prescriptive views of what made an award coherent. Students following 
CAT programmes appeared to have greater flexibility, although in the end the 
programme of study they chose had to be approved by the staff. However flex- 
ible the scheme might be in principle there was always some irreducible core 
of modules or credits which had to be completed, the remainder might be filled 
by elective or complementary modules. Some core modules might form part of 
several different routes, while some electives (like a modem language, IT, or 
Business Studies) might be appropriate for many routes. 

Qearly different insfitufions balanced the student and academic views of co- 
herence differently. Some were clear that coherence was a matter entirely of 
academic judgment, made by the staff, in which case the routes were pre-deter- 
mined, and offered limited or no choice. In other cases, it was, at least in part, 
determined by the student, in which case there was a wider element of choice. 
Some individuals and insfitufions clearly felt that students could bring their 
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own sense of coherence to a degree programme, and provided they could jus- 
tify a particular choice of subjects or modules, these might be permitted. 

Employer views of coherence were more difficult to obtain, and varied widely. 
Some wanted more emphasis on generic competences and several suggested 
that the learning outcomes had described too many 'academic' ideas. One em- 
ployer suggested that there was a need to "justify to students and employers" 
the inclusion of these learning outcomes. 

Consultation with employers in the development of degree courses was, how- 
ever, a very common practice in some subject areas, as were employer consult- 
ative groups which advised on curriculum content and in some cases provided 
examiners. There was cleariy employer involvement in the institutional devel- 
opment of some courses, although not enough in the view of some employers. 

Coherence, the cores and the clusters 

The notion of coherence was useftil in helping the groups to develop ideas 
about how learning outcomes might be clustered, and how core areas might be 
defined. It provided an alternative way to address the issue of grouping out- 
comes, avoiding the unhelpful separation of knowledge based outcomes from 
those relating to more practical skills and competences (a distinction which had 
proved to be a problem in assessing the outcomes). It also allowed outcomes 
which were common to all the subject groups to be identified, as common or 
generic outcomes, and outcomes which were specific to subjects to be separ- 
ated. 



5.5 Conclusions: Accreditation 

OveraU the groups took the view that it was neither necessary nor always 
possible to attach levels to all the outcomes of a degree programme. Levels 
based on assessment criteria could be developed and proved useful for some 
groups. It appeared that describing outcomes at the point of graduation was the 
most efficient and effective approach, and that there were then various ways in 
which student level and student progress towards that could be determined. 

The detemiination of credit value proved to be more difficult, but was best 
approached through discussion of the coherence and balance of the whole 
award. Notional time played a part in determining relative value but was not 
felt to be appropriate as the first or the sole determinant of credit value. 
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Accreditation: three 
approaches 

This chapter describes ond reviews some of the work on 
accreditation undertaken by the project subject groups. It 
describes three models based on general subject specific and 
core learning outcomes. 



6.1 Introduction 

The workshops and the materials provided a useful basis for discussion wit?::- 
and between the groups, but it became clear that those individuals who had no 
practical experience of working with CATs or with a modular structure had 
real difficulty in developing ways of Unking credits with learning outcomes. In 
consequence, almost aU the groups went back to their existing degree courees 
and allocated the learning outcomes to parts of the degree programme or, 
where they existed, to modules or units. They then calculated the credit rating 
of groups of learning outcomes by woricing out the proportion of the year 
which each part of the course, or module, occupied. For example: 

"for a course made up of six equally sized parts or modules 
in the first year, each module must carry a credit rating of 20 
credits, " 

The relationship here is cleariy between the course structure and the credits, 
and is not directiy between tiie learning outcomes and the credits. Attempts to 
develop alternative approaches yielded some useful ideas although they did not 
provide a wholly workable alternative. The following section describes three 
approaches: two developed by subject groups during the project, and a third 
based on the curriculum model developed by Alvemo College (USA). 

6.2 General Learning Outcomes 

The E)esign group identified five major design activities: 
research and analysis, 
planning, 

creating and making, 
communicating, 
evaluating, testing, deciding. 
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These were described as clusters of learning outcomes, but were effectively a 
list of ^general learning outcomes', since the clusters of statements which ac- 
companied then were used as assessment criteria. Each activity included a list 
of 'learning outcomes". There were, in all, 69 of these and the six related to the 
•planning' activity are shown in figure 6.1 below as an illustration. 



Planning 

To establish objectives and priorities, including personal aims. 
To plan a strategy which will lead to a design. 

To tackle many differing kinds of design thinking and practice at the same time. 
To order and record information. 

To manage their own time by plotting a way through required tasks and meeting 
deadlines. 

To have knowledge of how to find employment, including setting up in business 
or freelancing. 



fYgure 6. 1 - Design: Outcomes in Planning 

In practice, the group did not attempt to allocate credits to the learning out- 
comes but chose instead to use them as assessment criteria. For example Shef- 
field City Polytechnic (which already operates a CAT system) divides its full 
time courses into 6 units per year, each unit accumulating 20 credits. Credit is 
awarded to students on successful completion of each unit, that is for ''meeting 
the unit's requirements for directed learning". The 5 activities were weighted 
within each Design course, reflecting both the nature and stage of die course. 



BA Industrial Design 


Year 1 % 


Year 2 % 


Year 3 % 


research and analysis 


20 


20 


20 


creatingAnaking 


40 


30 


30 


planning 


10 


10 


10 


communicating 


10 


20 


20 


evaluating 


20 


20 


20 


BA Metalwork and Jewellery 


Research and analysis 


10 


10 


20 


creating/making 


50 


40 


30 


planning 


10 


15 


15 


'.municating 


10 


15 


15 


1 evaluating 


20 


20 


20 



Figure 6.2 - Assessment Weighting of Activities in 2 Courses 
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Figure 6.2 shows how three different institutions might use the activity head- 
ings (a-k), in different ways reflecting differences in the objectives, structures 
and ethos of their courses. 
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Figure 6.3 - Three Institutions and Cowmon Outcomes 

The structure has some advantages, specific outcomes included in a module 
could be negotiated between staff and students, and students would be clearly 
aware of the outcomes they needed to develop in each unit and those they 
needed to satisfy when undertaking specific pieces of assessed work. The 
presence of clear criteria would also make it easier for staff to assess any evi- 
dence offered for the accreditation of prior learning. 

A similar exercise was undertaken by the Social Science group, which used the 
learning outcomes to describe the requirements for progression from one year 
to another: some learning outcomes would be completed by the end of year one 
and others at year two and year three. 

In this approach the ^learning outcomes* do not themselves carry credit, they 
provide instead a set of assessment criteria against which the student can be 
judged to have satisfactorily completed the whole module. A student will there- 
fore receive either 20 credits or no credits. 
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6.3 Subject Specific Learning Outcomes 

The Environmental Science group defined in detail a body of knowledge 
which, was essentially the syllabus of one of the courses. This had eight major 
knowledge based headings: 



fundamental concepts. 



characteristics, elements and structures, 



environmental processes, 



systems functioning. 



complex inter-relationships, 



origins and history. 



multi-disciplinary nature, 



managing the environment. 



Each heading included a group of learning outcomes: for example, 
Environmental Processes. 

Learning outcomes: the student should be able to explain: 

- the range of physical processes which shape the land and 
transport sediment at the earth's surface,(learning outcome) 

- the basic hydrological cycle, 

- the process of photosynthesis (anatomical, physiological and 
biochemical), 

- the mechanism of coupling of energy yielding and energy 
requiring processes, 

- how environments in the past have conditioned the present 
day resources of mineral and coal 



The group described separately a set of learning outcomes which it described 
as "practical learning outcomes": 



Identifying appropriate equipment for lab and fitldwork 



Handling of materials 



Preparation of materials for experiment and analysis 



Developing ideas in a research project 



Field observation 
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Data analysis and presentation 



Use of computers, modelling and mathematical approaches 



Identifying and classifying organisms, soils and roctsS 



Experimental design 



Site evaluation and assessment 



Drawing up enviromental management plans 



They also pnxiuced a third categor>% of personal outcomes: 
Thinking 



Speaking 



Listening and responding 



Interpersonal skills 



Finding and reading source material 



Writing 



Developing independence 

It was not realistic to credit rate each learning outcome, since the demonstra- 
tion of the knowledge based outcomes necessarily involved some of the p'^cti- 
cal and personal outcomes and vdce versa. On^ possible approach was to 
allocate credits to each area of knowledge, and use the personal and practical 
learning outcomes as assessment criteria for the learning outcomes. 

a student should be able to explain the major classes of pesticide 
and their primary modes of action. 

The assessment of this might involve the student in a range of different acti- 
vities designed to demonstrate the practical and personal skills, 

carry out laboratory work. This might involve experimental design, 
laboratory safety, data analysis and computer analysis and 
presentation of data, for e uimple - present his/her findings to a g^oup 
in a 15 minute talk. 



It would not be difficult to consider each learning outcome from the knowledge 
section in turn, decide on an activity or activities which were appropriate to 
develop it and work out how it was going to be assessed. Each outcome would 
then have associated with it a set of detailed criteria describing exactly what 
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the student had to do to provide evidence of achievement of that learning out- 
come. 

This approach avoids using the course structure as a basis for allocating credits, 
instead it uses a series of so called areas of knowledge. This is an interesting 
alternative, particularly in the way it relates the 'harder' and 'softer' learning 
outcomes together, effectively ensuring, (or acknowledging?) that one is not 
acquired witiiout the other, 

6.4 Core Learning Outcomes 

The Alvemo College (Milwaukee, USA) ability based curriculum has been de- 
scribed and discussed extensively in the literature and was offered to the 
groups as a possible approach^^. It is included here since it provides a com- 
plete contrast to the models described above. 

The Alvemo curriculum is described as ''ability based and outcome oriented". 
It is entirely organised around eight key abilities at six levels, the abilities are 
described as follows: 

analytical capabilities; 

woiicable problem solving skills; 

facility in making judgments and independent decisions; 

facility for social interaction; 

responsibility towards the global environment; 

effective citizenship; 

aesthetic responsiveness, involvement with the arts. 

The six levels were derived from the Bloom taxonomy, but are seen as devel- 
opmental stages rather than absolute levels, a student may therefore demon- 
strate an ability at level Uiree without first being required to demonstrate levels 
1 and 2. Levels 1 and 2 are regarded as a useful learning route however. 

In order to graduate (Bachelor's degree), a student must have 32 units, that is 
s/he must have demonstrated all eight abilities at levels 1-4, plus a further eight 
units above level 4, and at least one imit at level 6. Subjects (Alvemo offers 44 
including Chemistry, Music, English and Nursing) are offered through the abil- 
ities, so for example Business and Management offer units which include prob- 
lem solving at Levels 1-6, and valuing at Levels 1-3. However, not all subjects 
offer a full range of abilities : business, for example, does not offer units which 
include aesthetic responsiveness, so a student must complete a unit ^^om an- 
other subject in order to achieve this ability. The degree is thus centred on the 
abilities which are achieved and demonstrated through the subjects. 

The College believes that all the abilities can be developed and assessed, they 
are therefore a requirement of the degree, a student cannot "fail" one and 
graduate, however good s/he may be in the others. This requirement looks 
rather hard to tiiose used to the UK model of compensation in assessment, but 
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set in the context that the College believes and supports the idea that the abil- 
ities can be developed it sets a high but not impossible standard. The College 
degrees arc accredited by State and National Bodies and employers are actively 
involved in reviewing the abilities and in the assessment of students. 

An Alvenio style model could be used as the basis of a structure for accredit- 
ing learning outcomes, assuming for the sake of simplicity that there were eight 
core learning outcomes (comparable to the eight abilities) then each of these 
might carry 15 credits at each CAT level. Tne learning outcomes for each 
' subject could then be clustered around these eight areas and a student would 
achieve the 15 credits by demonstrating each learning outcome. 

6.5 Recommendations - An alternative 
structure 

6.5.1 The Findings 

The three approaches described above arc based on: 

general learning outcomes, vocationally or practically related, 
subject specific learning outcomes, 
core (all graduate) learning outcomes. 

The learning outcomes originally described by the groups had contained 
examples of each of these types of outcome. It also appeared that many of the 
problems the groups had experienced in linking credits with outcomes had con- 
cerned whether to start with the general, the specific or the core. The groups 
. had also had difficulty in detemnining the relative weight (in terms of credits) 
of the three types of outcome, this was related to the difficulty some of them 
had in describing what they meant by coherence in a degree programme. A 
simple solution to the difficulties might be to create a model of the learning 
outcomes, in which the relative contribution of general, core and specific out- 
comes could be varied. This would allow the relative proportions of the differ- 
ent types of outcome to be clearly determined and this would help in linking 
credits to the outcomes. 

The findings on the assessment of learning outcomes and the responses re- 
ceived from employers and students, also contributed to the development of a 
possible model. Most of the employers consulted responded positively to a set 
of what were described as common learning outcomes, which were not subject 
specific. These had been arrived at by looking for learning outcomes which 
were common to all five subject groups. Employer responses to the subject 
specific learning outcomes were less positive. Many employers stated that they 
were not interested in these since they recruited graduates from a wide range of 
subjects. Other employers suggested \h^* they actually recruited graduates 
using a set of critena which were not subject specific, and this is supported by 
much of the literature on graduate recruitment. 

The student consultation similarly suggested that students were able to describe 
a number of common learning outcomes which were ciihci vocationally related 
or were related to their personal and social development. Few students de- 
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scribed subject specific learning outcomes, and those that did, tended to pro- 
vide very general information. 

The outcomes described by academic staff were by contrast largely subject 
related, although they did include both vocationally specific outcomes and core 
outcomes. 

6*5*2 A 'curriculum modeF 

In the ligl:it of this there appear to be three possible models for accreditation, 
beginning from outcome clusters based on: 

general outcomes 

subject knowledge 

core outcomes. 

A Model Based on General Outcomes: Design 

The Design group model begins with five practice based groups of outcomes. 
The weighting of these might vary according to: 

the nature of the course ( e.g. two or thiree dimensional design) 

the institution 

the stage of the course 
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Figure 6A - A General Outcome Model 
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The model might also include a set of core outcomes, common to all graduates. 




Figure 6.5 - Incorporating the Core 

The core would not be taught or assessed separately, but outcomes would be 
ar hieved and assessed through the five practice based areas. Each of the prac- 
tice areas includes a wide range of outcomes, some of wWch were knowledge 
based, and some skill based. 

Relating the model to credit values could be achieved by calculating relative 
values. For example (from Fig 5.1) "Creating and Making" was weighted as 
40% of year 1 and would therefore represent 48 credits at level 1. The achieve- 
ment of that credit would be dependent on the student satisfying the outcomes 
listed within that heading - e.g. 

manipulate design forms with a high degree ofskilL., 



demonstrate an understanding of the use of colour through... 



use a range of visual processes ^photography^ drawing) to describe 
design ideas. 

Precise assessment criteria for these outcomes were developed through dis- 
cussion with student groups. 

A Model Based on Knowledge: Environmental Science 

Tne Environmental Science group described 8 major areas of knowledge, 1 1 
practical outcomes, and 7 personal outcomes: 

The relative weighting of the knowledge areas might again vary between 
courses and stages of a course. 
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Figure 6,6 - A Knowledge Based Model 

The pereonal outcomes would form a part, or aU, of the core. The practical 
outcomes w .^uld represent a second dimension of the model, which might be- 
represented as follows. 




field work competence 
laboratory v/ork competence 
information handling competence 
experimental design competence 
classification competence 
' etc. 



Figijre 6.7 - A Second Dimension 
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A Model Based on Core Outcomes: Alvemo 

The Alvemo College model begins not from general outcomes or knowledge, 
but from the core outcomes or "Key Abilities" which are effectively an ex^ 
panded core. These are: 

analysis, 
problem solving, 
decision making, 
social skills, 
environment, 
citizenship, 
aesthetic. 

Credit value attaches primarily to the Abilities rather than to the subjects, 
which provide a vehicle for the development of the abilities. 



Figure 6,8 - A Core Based Model 
6.5.3 Credit without time 

The use of the ^curriculum model' makes it possible to detemainc relative 
credit weightings for different groups of learning outcomes without reference 
to time. The two groups (Design and Environmental Science) considered the 
effect this would have on the delivery of the course. 





8 key abilities, each 
wltii a credit value 



Subjcct/s 
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Design 

One of the Design courses was currently modular, with students completing 6 
modules per year, each carrying 20 credit points. The Design course was based 
on a series of projects, students were provided with a *brief ai.^ required to 
produce a d^esign to meet its requirements. When the credits were related to the 
outcomes, rather than to the course modules, the modules simply provided a set 
of learning opportunities (the projects) through which the students demon- 
strated that they had achieved the outcomes. This would place much greater 
responsibility c«i the student to make certain that s/he was providing evidence 
to meet the requirements of each outcome. It would require staff to play a more 
facilitative role to support and advise students, and provide them with feedback 
on progress. 

Environmental Science 

The Environmental Science course structures were very varied. Some courses 
were modular and offered students a range of options, while others were struc- 
tured as a linear programme without any options. Two of the courses did not 
use a credit structure, but it was relatively easy to see how the achievement of 
outcomes could be related to the credit scheme. Some elements of the course 
were designed to provide opportunities to demonstrate particular outcomes re- 
lated to the eight areas of knowledge, and it was not difficult to relate these to 
credits. 

One course was modular and had recently introduced learning outcomes into 
module descriptors. Credits, however, were related to the size of module, 
measured in *notional learning time' rather than to the achievement of the out- 
comes. Further, since it was not clear that the assessment methods used in the 
modules actually related to the outcomes themselves, it was not clear whether 
there was any direct relationship between the identified outcomes and the 
credits awarded. 
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This chapter presents the overcdl findings of the work in three 
parts. It outlines the intentions, conclusions and achievements 
of the project; comments on the findings about leoming 
butcomes; and the process of defining them. 

7.1 Introduction 

7.1.1 Objectives of the project 

The purpose of this project was to examine the feasibility of describing a de- 
gree in terms of its outcomes - what a graduate can do, know and understand. 

It began from the premise that a degree is currently described in temns of the 
process - three years full time study - rather than achievements, and that, as a 
result, notions of quality are based on the process, rather than the outcome. 
This approach may be increasingly difficult to sustain as many of the elements 
of the traditional process are removed or reshaped in the move from an elite to 
a mass higher education system. It can no longer be assumed, for example, that 
most higher education operates with low staff-student ratios, a homogenous 
student body or single subject based three year full time degree courses. A 
continuing fall in the unit of resource, coupled with rising student numbers will 
continue this pressure. 

The project proposed that a stronger focus on outcomes would enable quality 
assurance processes to be more transparent and rigorous. WhUe it may be 
possible to achieve the same outcomes by other processes, if the outcomes of 
the traditional model are not adequately described, there is no way of assessing 
the impact of changes on quality. Adequate ways of describing outcomes arc 
therefore essential to the defence of quality in a rapidly changing context. 

It was also believed that the description of outcomes alone would not solve the 
problem, and that it was essential that those outcomes could be assessed, and 
recognised in systems of accreditation. Credit would thus be more directly 
linked to explicit achievement. A further benefit of the approach would be in 
the increased ability to recognise and accredit prior learning, and learning 
undertaken outside formal classroom settings. 

The project's aim was exploratory and developmental. It sought to find ways 
of recognising the full range and diversity of achievement which existing de- 
gree programmes represent, not to narrow or constrain them. It therefore set 
out to investigate a range of approaches with practising academics, rather than 
to impose a pattern or methodology. 
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7.1.2 Key Conclusions 

The project woric points to the following key conclusions: 

• it is possible to describe the outcomes of higher education more 
explicitly, although they cannot be expressed in simple *can do' 
statements, and, in a complex and changing environment, such definitions 
will never be complete or fixed; 

• descriptions of learning outcomes in higher education cannot be 
expressed as a single set of 'national standards' of the kind developed for 
national vocational qualifications, since higher education exists to meet 
the needs of a variety of client groups and a range of social, economic, 
scientific and culmraJ needs, and properly embodies a range of different 
cultures and value systems; 

• it is necessary to develop processes within each instimtion to link 
outcome definitions with quality assurance, since the authority to define 
the purposes of degree programmes rests with the chartered institution, 
rather than with any national agency; 

• the principal benefit in an outcomes led approach lies, therefore, in 
providing a focus for staff, students and employers to examine more 
cleariy what they are seeking to achieve, and in enabling them to 
contribute actively to the development of a common understanding of the 
namre and purposes of higher education and of specific programmes and 
awards; 

• the process of defining and accrediting outcomes can best be seen in 
terms of a four stage cycle: 

preliminary definition by academic staff, 
design of an assessment strategy, 

production of a 'curriculum model' v. Wch weights the various 
types of outcome - core, subject specific, general and other, 

allocating credit value to groups of outcomes; 

• a range of materials and approaches exist to help in the definition of 
outcomes. These include models based on objectives, subject knowledge, 
Uie discipline and competence. No single one has been demonstrated to 
be ideal and there may be advantages in using more than one; 

• if quality is to be assured it is important that the assessment and 
curriculum modelling stages of the cycle are carried out before the 
allocation of credit value; 

• the definition of outcomes should not be separated from an investigation 
of assessment since outcome definitions which are not assessable are not 
of any practical value; 

• an outcome led approach requires staff to develop and use methods of 
assessment which measure achievements direcUy, but current assessment 
practice tends to neglect these questions of validity in favour of 
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reliability, and many academic staff lack experience of appropriate 
approaches to assessment; 

• employers are, in general, more concerned with the "core" learning 
outcomes of higher education than with general or subject specific ones; 

• students can play a particularly valuable role in the development of 
approaches to assessment, and this dialogue can itself help to increase 
their motivation and advisement; 

• the definition of learning outcomes is considerably helped by the use of 

^ss-institutional subject groups, which provide a broader contrast of 
perspectives than is possible in a single institution. Opportunities for this 
kind of staff development appear to be rare. 

7.1.3 What was achieved? 

The projea has helped to "flesh out" ways of describing learning outcomes in 
higher education, to clarify who can and should be involved in their develop- 
ment, and in what ways, and explain how they might relate to NVQs. It 
achieved two things: 

• it tested and refined a collaborative model for the development of 
learning outcome statements, working from within the academic culture, 
rather than by imposing managerial models from outside. It tried to 
develop outcome definitions through an iterative and collaborative 
process, using academic subject specialists, and seeking a consensus. It 
sought to capture the essence of what it means to be a graduate in 
engineering, or in design. Tne project used a very simple experiential 
approach, based on the beliei that the solution lay in helping practising 
academic staff to make their tacit knowledge and understanding explicit 
and public, rather tha.. through the simple application of external theory; 

• it produced a quality development model which links the definition and 
assessment of outcomes with an approach to coherence and a credit 
framework. This has important implications for the quality assurance of 
all learning in HE but especially for learning which is not part of a 
course, Gike some work based learning) and for the accreditation of prior 
learning. 



ERiC 80 



Findings 



7.2 Findings about Learning Outcomes 



7.2.1 Linking the Elements 

In many respects the most significant finding of the project is the importance 
of linking the description of learning outcomes with the development of an 
assessment strategy and with a credit accumulation structure. 




Figure 7.1 - The Elements 

This relationship seems simple but proved difficult, because some aspects of all 
three notions are unfamiliar to many academics. The superficially siinpl" steps 
required clear and explicit definitions of principles and practices which were 
often implicit and tacit, and which embodied deep seated cultural values. 
Learning outcomes were, for most staff, the outcomes of courses, and it was 
easier to describe the course in terms of what was taught, than to describe what 
students were expected to achieve. Working in cross institutional groups 
helped, since the different experiences of group members showed that the 
learning could be packaged in different ways, and at different stages and that, 
although there were differences in courses, there was much in common in 
terms of what students were expected to achieve. 

Reviewing methods of assessment and examination questions was also useful, 
but approacliing the problem entirely through a review of assessment, was 
limited by the nan*ow range of assessment methods in use and by the evidence 
that much current assessment was more closely linked to successful completion 
of the course, than to demonstrating the achievement of outcomes. Overall, the 
description of learning outcomes was difficult to disentangle from notions of 
courses and objectives. Staff v/ith some experience of APL, of assessing work 
based learning, or of CATS schemes had an advantage here. 

The groups recognised the need to identify a variety of kinds of outcome, but 
fr>Mnd that it was important not to divide learning outcomes into groups which 
separated factual knowledge from competences, practical skills or personal 
skills, since this posed problems when attempting to assess the outcomes separ- 
ately. 
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7.2.2 The Importance of Assessment 

The project's work on assessment was very much more important than antici- 
. pa ted at the start. Just as many staff found it difficult to express what the 
intended outcomes of their programmes were, so the methods of assessment 
they used often failed to measure what they were seeking to achieve. 

The work on the assessment of learning outcomes proved easier for some staff 
to tackle and was cleariy work which they regarded as of immediate personal 
relevance. Previous expertise in assessment was very varied and what was re- 
garded as innovative in one group was seen as relatively mundane by others. 

The work on assessment stimulated more short tenn changes in thinking and 
practice than did the development of the learning outcomes themselves. In 
retrospect it is clear that, had the workshops on assessment been provided at an 
earlier stage , they would have focused the descriptions of learning outcomes 
more sharply and helped to avoid some of the less productive debates. 

Problems 

The requirement to assess the outcomes which had been described exposed 
some serious problems with assessment generally: 

• for some academic staff, experience of assessment was limited to a small 
range of relatively traditional methods, which were not applicable to some 
of the outcomes they had described; 

• most staff gave undue weight to the reliability of assessment, at the 
expense of validity. 

U was necessary therefore to make some imaginative Icps in the development 
of different approaches to assessment. The project has only touched on these 
and there is a clear need for further work. 

An Experiential Approach 

The project used a simple experiential 9.trategy, providing advice and informa- 
tion and encouraging staff to develop ways of assessing the learning outcomes 
they had already described. This approach, which required staff to apply the 
critical skills which they used in the study of their own subjects to the develop- 
ment of alternative forms of assessment proved to be very successful and did 
not require an extensive or complicated external staff development programme. 

Assessment strategy 

The solution to the assessment problems encountered within the projea docs 
not lie simply in the adoption of a number of new assessment methods, but in 
the development of a coherent assessment strategy. 

The project experience suggests that changes in assessment need to be accom- 
panied by a consensus amongst the staff about what was being assessed, not 
just about how to do it. The adoption of a different approach to assessment by 
one member of staff led to confusion amongst students, who felt that they were 
being asked to produce one sort of work for one member of staff and some- 
thing different for the rest. 

as 
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The student input to the projea was also important at this point since it demon- 
strated that students were able to work with the learning outcomes to develop 
and agree standards and criteria for assessment. This suggests that the out- 
comes could be used to develop greater learner independence, as well as more 
active and learner centred approaches to learning. 

The assessment pilots made an important contribution to the project since they 
demonstrated that it was possible to assess some of the learning outcomes 
which staff did not currendy assess, and which some people had regarded as 
impossible, too time consuming, or too subjective to assess. Students also sug- 
gested that they were not always clear about the criteria being used by the staff 
in assessing their work. TTiis lends weight to the view that clear descriptions of 
learning outcomes would be of benefit to students. 

Grading 

Overall the description of learning outcomes does suggest that if a clear link 
can be made between outcomes and assessment the requirement to grade a 
student may be redundant. However, the question of whether, and how, to 
grade the assessment of learning outcomes remained unresolved. Some staff 
cleariy felt that it was unnecessary to grade all work, and wholly impractical to 
grade the assessment of some learning outcomes. There was some reluctance to 
address the question of grading in relation to a degree, although the concept of 
an undivided Honours degree was discussed. Such decisions need to be made 
by course teams as part of an overall decison on assessment strategy, this needs 
to reflect the culture of the institution and the course, 

7.2.3 Outcomes and Credits 

T?ie project work suggested that it is possible to build a link between outcome 
and credit, but that this can only be achieved if the outcomes can be assessed. 
Since outcomes in themselves have no credit value they need to be clustered 
into groups of outcomes which can either be constructed by dividing the final 
award into parts, or by assembling 'units' by clustering outcomes directly. 

The ability of the academic staff in the project groups to relate outcomes and 
credit value was limited by a lack of experience of credit accumulation 
schemes and by the lack of a general consensus about key terms like co- 
herence, balance and level. As a result, although it was possible to describe 
learning outcomes, it was not easy to move directly from these to a notion of 
relative value, and thus to a structure for the award of credit. Staff Uierefore 
tended to relate their outcomes directly to credits, using traditional course 
structures and ^notional*, or actual, learning time as a basis for determinig 
credit value. 

The problem with this approach is that in bypassing assessment, a crucial 
dimension of quality assurance is lost, because credit is being allocated to 
teaching processes and not to learner achievement. Learning outcome state- 
ments tended therefore to be indistinguishable from course objectives, and any 
learning which was not course based had to be calibrated against a fonr il 
course. This effectively kept the course at Uie centre of Uie structure and 
preserved the paradigm of a process, rather than outcome, based degree. 
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7.2A Outcomes and Coherence 

In an attempt to address this issue, the project suggested that the link between 
learning outcomes and credits is best made through an intermediate structure, 
which provides a means of determining the relative value of leaming outcomes, 
without recourse to teaching time. This proposed structure can be seen as a 
model of coherence in an award, or as a 'curriculum model' which would rec- 
ognise the existence of several kinds of outcome within the same award. Figure 
7.1 shows a way of presenting this. Individual degrees would have the three 
elements in varying proportions. 




Figure 7.2 - A Model of Coherence 

The employer and student input proved useful at this stage since it made clear 
that both groups were concerned with a relatively small but overarching group 
of 'core' graduate outcomes, which applied to all degrees regardless of subject. 
There was also evidence from the groups that a consensus on the core might be 
fairiy readily achueved, since there was considerable overlap between some of 
the outcomes described by the groups, and a number of common strands can be 
found in definitions of 'core' used elsewhere in education and training.^^ Th'^ 
'core' would provide a broad description of the outcomes which employers, 
students "nd broadly speaking, society in general, expeaed a graduate to 
achieve. The national experience of the development of core skills and com- 
petences showed that they need to be integrated into programmes of leaming, 
not added (xito them, and that they needed a subject, or an applied context in 
which they could loe developed and assessed. The core needed therefore to sit 
at the centre of the model. 
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The project exposed the debate on the meaning of level, coherence, breadth and 
depth in relation to an award. Here the processes of collaboration and consult- 
ation have helped to show that these questions are closely related to both the 
values of Uie subject/discipline and the mission of HE itself. 

7,2,S Types of Outcome 

In view of the diversity of cultures and value systems embodied witiiin higher 
education, a model of coherence needs to be able to recognise more tiian one 
kind of outcome. The balance between Uiese would vary between awards ai?-^ 
institutions. There would appear to be at least four types' of learning outcome: 

Core Learning Outcomes 

These represent those qualities generally identified wiUi Uie notion of *a grad- 
uate' and must be achieved by all graduates regardless of subject. 

Examples produced during the project which would fall into this category 
might include: 

ability to communicate complex information to a variety of audiences 



ability to make independent decisions and judgements 



time management 



ability to accept and learn from criticism 

Some employers proposed a rather different kind of core outcome: 

the ability to understand the reality of business problems and the 
relationship ofspec fic tasks to overall economic objectives 

The notion of core outcomes was not contentious in itself and there is probably 
a broad consensus about them in higher education, albeit not an explicit one. It 
seemed likely that in many cases these outcomes were achieved by most grad- 
uates, and that presenting them in this way simply fomialised tiiat. However, 
assessing and awarding credit to them would be a way of making clear to 
graduates and employers, that tliey were an essential, rather than optional, part 
of tfie degree. This would be consistent wiOi tiie ideas on assessment developed 
in Chapter 4, where it was clear that outcomes such as tiiese were not inde- 
pendent of the subject, or learned in separate classes (called, for example 
•problem solving'), but were required outcomes of several different areas of a 
students work. A student might therefore be required to present several differ- 
ent pieces of evidence to demonstrate her/his problem solving ability. 

The relative size of the core might vary between degree programmes. At its 
extreme, the model presented by Alverno CoUege , presents Uie whole degree 
in terms of core outcomes, with the subject material being built around this. A 
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more common model would have a core at its centre, with elements of the core 
assessed in different parts of the programme. 

There are a number of models of core which might be appropriate. The project 
derived a simple set by mapping the common elements of tJie material pro- 
duced by the separate subject groups, but several alternative, and more care- 
fully developed, models exist. 

Subject Specific Learning Outcomes 

These reflect the values and culture of the academic discipline and are closely 
related to the specific subject of the degree. They describe the qualities implicit 
in tlie notion of an 'engineer' or a ^historian'. 



knowledge of the present day functioning of major social institutions. 
This is a personal quality which arises from the nature of the Social 
Sciences. A specific example would include the ability to evaluate a 
report on institutional change within the Church so as to highlight 
those assertions which are insufficiently underpinned by adequate 
evidence of cogent argument 
- Social Science 



to apply a visual vocabulary, to relate shapes and forms and to 
express them 
- Design 



abiUty to calculate energy values and forces present when 
elctromagnetic induction is produced by interacting electric and 
magnetic fields 
- Engineering 



exlpain the priorities offered by "green" and conservation opinions 
- Environmental Science 



The description of subject specific leaming outcomes again depended on the 
nature of the subject or subjects, and on the awarding institution. These might 
form the largest category of outcomes in a single subject degree, but a smaller 
proportion in others. 

The subject outcomes were related to the title of the award and might reflect 
single, joint or multiple subjects. These outcomes related to the study of the 
subject, and included subject, or discipline, related concepts and methods of 
enquiry as well as a range of types or categories of knowledge. In general, 
these outcomes would be common to degrees with the same name offered by 
different institutions, although degrees would vary in how many were included, 
and how they were clustered for assessment and accreditation. There were sig- 
nificant differences of this sort within the project subject groups. 



21 Note 4 identifies some of these. 

22 See Chapter 3. 
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General Leaming Outcomes 

These would reflect the underpinning skiUs needed in empioymenl and might 
include requirements to practice in particular occupations. They might be com- 
mon to several different subjects but not to all graduates, and might include 
leaming outcomes which crossed several different subject areas. 

This was also an area where NVQ units in specific areas could be included and 
count for credits within a degree programme. 

understand common financial terms 
' Design 



safe and correct handling of materials in the laboratory and tfie field, 
e,g, handling micor-organisms safely using aseptic techniques 
- Environmental Science 



ability to decide between alternative courses of action in an 
engineering setting 

assess energy saving methods to achieve minimum operating costs 
select and reject manufacturing processes to produce a specific 
component most economically 

recommend techniques to minimise failure/maximise life of materials 
and components 
- Engineering 



knowledge of a range of teaching techniques which can be used and 
adapted to communicate different types of material 

...specifically to lead a group discussion on changing conceptions of 
the society/ environmental relationship; ability to direct a role play 
exercise on the topic of Journalists^ right to protect their sources; 
ability to plan a series of seminars ..about the peresentation of women 
in the British cinema 
- Social Science 



These outcomes also varied according to the subject and to the institution. In 
some cases the degree was effectively a professional qualification, or a precur- 
sor to one, and the outcomes reflected the need for the student to be able to 
practice in a profession, as well as demonstrate an academic knowledge of the 
subject. In others there was relatively little, or no, weight given to vocational 
outcomes, or they were U"eated in very general terms. This was the point at 
which the place of NVQ awards and units became clearer, since they might 
well form all or part of the vocational outcomes of a degree. The fact that they 
would be externally defined and might be assessed in differrent ways from 
other outcomes need not present a problem within Ihe proposed framework of 
coherence. 



Other leaming outcomes 

There might also be other types of outcome. Individual institutions might wish 
to specify other outcomes in this category, related to their particular institu- 
tional mission, adding, for example, languages, enterprise or environmental 
outcomes to all their degrees. 
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Chapter 6 presents a possible approach to award design based on this model. 



7.2.6 A QuaUty Cycle 

The process can thus be seen to have four stages, in a logical order, and im- 
plying a continuing cyclical review which enables each part to be reviewed in 
the light of experience. The four stages are show in Figure 7.2. 



stage I 
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outcome 



stage 4 



credit rating 



stage 3 



curriculum 
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stage 2 
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Figure 7.3 - From Outcomes to Credit a Quality cycle 
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7.3 Findings - V/ays of Working 

7.3.1 Working with Cross-Institutional 
Subjetit Groups 

It would in many ways have been easier to achieve the project's primary pur- 
pose by appointing a single subject specialist for each area, rather than to use 
groups of sf zii. However, the Project Steering Group agreed to use cross in- 
stjsudo'-.ai groups, to provide cross fertiUsation and diverse perspectives on 
each subjea. Tney also beUeved important to test whether such work could be 
car.ied out effecUvely by pracUsing academics rather than educaUonal re- 
jearchers..7^.i project found that: 

• the groups were intrinsically valuable in providing an opportunity for 
experiential staff development, of a kind rarely achieved in the occasional 
seminar. The variations between the member of the groups, the 
instituUons they represented and the courses they taught were vast, and 
developing an understanding of these approaches was both useful for the 
staff concerned and valuable for the project. Hiis proved a genuine 
manifestation of the "academic community" which, in other ways seemed 
notably absent from the institutional work. 

• the groups provided an opportunity, which did not previously exist, to 
discuss at length and in detail matters Uke course content, approaches to 
teaching and learning, course design and assessment. Such opportunities 
are normally available only to external examiners and those invited to 
take part in peer review and validation exercises. 

• the consultative document stimulated debate and development woik in 
institutions not directly involved in the project, and the responses 
stimulated further debate and development in the project institutions. 
Although the document was acknowledged to represent 'woric in 
progress* rather than a completed model, it elicited a wide range of 
approving, constructive and critical comment. 

Characteristics of success 

Although the groups and seconded staff achieved most of the objectives set. il 
was clear that some were more successful than others. The characteristics ol 
the former were: 

• previous experience of either a modular course or a CAT credit system 
(Some simply required a longer period of time to assimUate all tiie ideas) 

• experience of course development, or of a modular or credit based system 
to which they could relate their work. 

• a position within their department which provided an overview or context 
for the woric and its implications. (Some individuals lacked a clear view 
of their own overall course structures and brought a rather narrow subject 
perspective to the thinking of the groups). 
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• a clear understanding of credit transfer, or the CNAA CAT credit system 
and its implications. (When attempts were made to link the learning 
outcomes to credits, some groups and individuals realised that they 
needed to adopt a completely new approach to thinking about courses and 
to how they assessed students. Others tried, with little success, to fit the 
learning outcomes to existing course structures, resulting in enormously 
complicated and confusing lisLs of credits and outcomes). 

• availability of adequate time. (Despite careful agreements drawn up with 
institutions, some staff came under severe pressure of work which 
prevented them from participating fully). 

• an ability to make a conceptual leap to view their subject from a new 
perspective. (Not all staff appeared able to do this). 

• adequate institutional support and internal networks for communication 
and debate. (With some exceptions, group members worked in isolation 
in their institutions, receiving little or no support from colleagues and 
having, or taking, little opportunity to discuss their work with colleagues). 

Problems 

Overall the strategy of creaUng small subject groups worked well. It produced 
the materials required and was clearly of benefit to the individuals taking part. 
However it failed in some respects to have the impact sought. The ideas de- 
veloped and discussed in the groups did not penetrate successfully into the 
iastitutions they came from, and the opportunity for coUeagues to make a criti- 
cal input, and for the ideas to be generally shared was therefore lost. There are 
probably two reasons for this, the first was tlie failure of Uie project to establish 
and sustain links with senior management and with institutional quality assur- 
ance procesKs. The second was a failure of the project to create links with 
existing staff development structures in the instituUons and with parallel work 
un 'enterprise' which was ongoing in several of the institutions during the pro- 
ject. It may be that, as a result of recent developments in quality assurance and 
staff development, the strategic significance of these ideas would be more 
cleariy recognised now than it was at the beginning of the project. 

Interestingly, it would appear tliat the most acUve development of the project 
ideas has been in institutions which expressed initial interest, and maintained 
contact with the project, but were not selected to participate. By contrast, it 
appears that most of the institutions which took part in the work have not 
developed the ideas further. 

The consultation succeeded in stimulating discussion about the work and at- 
tracted a wide range of responses and opinions. The majority were positive, 
welcoming the thrust of the work and providing constructively critical com- 
ments on the learning outcomes. However, it was clear that many readers did 
not understand the purpose of describing learning outcomes and expressed 
fears about narrowness, reductionism and 'vocafional drift'. 

Surprisingly, Uiere was considerable support for the 'ad hoc' structure of levels 
which had been developed as a crude means of presenting the stages in work- 
ing out the learning outcomes, and was presented in What Can Graduates Do?. 
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7.3.2 Working with Students 

Ihe student workshop model was undoubtedly a success. It required students to 
respond individually and in groups to key questions, and to list and agree the 
learning outcomes they expected toAiad achieved. The findings suggest that the 
workshop is a robust and cost effective strategy which could provide a simple 
means of sampling student perspectives on learning outcomes. 

Many students welcomed the opportunity to discuss their views of learning 
outcomes and to express what they were seeking fiom their courses. Some of 
the academic staff working on the project expressed surprise iliat their students 
had such a wide range of views on their learning and were surprised at Uie 
depth and detail of the responses. Several staff reported the findings of the 
workshop as being valuable in terms of curriculum design and course develop- 
ment. 

The outcomes described by students were much broader than those described 
by the staff, almost half the outcomes which students anticipated or felt they 
had achieved were in the pei^sonal and social area, whereas only one quarter of 
the outcomes described by staff were in this area. 

The assessment pilot work also demonstrated the value of involving students in 
the discussion of outcomes. Students found the clarity of tlie outcome state- 
ments helpful in understanding what was expected of them, and the focus on 
assessment helped them to be clear about what constituted relevant and appro- 
priate evidence of achievement. 

7.3.3 Working with Employers 

The employer consultation provided some useful feedback on the leaining out- 
comes themselves and on employer views of the project generally. The latter 
were all very positive and constructive. However, the overall response rate is 
disappointing, although perhaps unsurprising. It is paiticulariy unfortunate that 
the majority of responses were from personnel staff rather than line managers: 
several covering letters suggested that line managers were *too busy' to con- 
tribute in this way. 

The most important finding from this exercise was that the majority of em- 
ployers welcomed the general learning outcomes but ignored the detailed sub- 
ject learning outcomes, many respondents wrote that they recruited graduates 
of any discipline and that therefore the subject specific learning outcomes were 
not relevant to Lhem. This was surprising, since many of the learning outcomes 
identified under the subject specific headings document were very general and 
included broad generic qualities like communication, planning and group work. 
It was also disappointing since the subject specific outcomes had been specifi- 
cally written to indicate to employers that some degrees developed learning 
outcomes with wider applications than the study of the particular subject. 
Nevertheless, tiie detailed responses which were received were helpful and sev- 
eral included lengthy comments on the material as well as including further 
information from their own experience. 
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Chapter 8 



A Model for Quality 



Previous chapters have outlined the project aims, processes 
and findings. This chapter draws on all this material to present 
a model for development in individual institutions. It proposes 
a Jour stage model linking learning outcomes with quality 
assurance processes. 



8.1 The Model 



The model shows how learning outcome statements might be developed and 
used in higher education institutions. It describes who might be involved at 
each stage and how outcomes link both to credit structures, assessment and 
teaching systems. 
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Figure 8.1 -A Model for Quality 
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Stage 1 - Description 

The first stage is concerned with describing the outcomes of a degree course. 
This may be done on the basis of an existing degree, or by describing the 
intended outcomes of new programmmes. It is not necessary for all the out- 
comes to be expressed in the same form, and indeed this may be a disadvant- 
age, since, within higher education, and single degree programmes, there are a 
range of purposes and value systems. Thus the form of outcome statements 
related to core skills may be different from general or subject specific ones. 

The model proposes that, in the first instance, description should be the respon- 
sibility of academic staff, since it is they, rather than external agencies who are 
resposible for standards in chartered degree awarding institutions. The project 
experience suggests that this will require an iterative and collaborative process 
which seeks to make more explicit the many implicit outcomes expected of a 
graduate of a given programme. While the process needs to be institutionally 
based, it can be effectively facilitated through collaborative woilc by specialists 
from different institutions, or from different parts of the same institutioa Hie 
procevSS is complex, and the outcome descriptors are unlikely to be simple *able 
to...* or behavioural statements. 

Chapter 2 offers examples of how the projea groups worked and of the sorts 
of learning outcome statement which they developed. Three main approaches 
are recommended, based on one or more of the following: 

Subject Knowledge 

Discipline 

Competence 

The first of these might usefully call of one or several of the existing published 
models for structuring professional knowledge, which provide a helpful start 
point to build indivudual models. Some standardisation of language is also an 
important step, and structures like Bloom's taxonomy proved useful in develo- 
ping a consensus on the meaning of words like "understand" and "explain". 
The second approach called for the elicitation of the tacit values of the disci- 
pline. The third approach involved the use of techniques like ftmctional ana- 
lysis to develop a better model for describing vocationally related outcomes. 



Stage 2 - Assessment 

If learning outcome statements are to have any function it must be possible to 
assess them in some way (although this must not be taken to imply that all, of 
even most, outcomes arc susceptible to crude testing). The project showed 
clearly that the requirement to assess outcomes was important in three ways: 

• it clarifies for s^aff and students what the student is required to achieve; 

• it forces staff to examine the validity of assessment methods by relating 
assessment directiy to outcome. The projea experience suggests that staff 
often become unduly concerned with questions of relaibiUty of 
assessment, at the expense of validity; 

• it provides an iterative feedback loop, to enable staff to refine the 
outcome statements themselves. 
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The student input may be particularly helpful at this stage of the cycle, since 
the students have a very direct concern with the fomn and process of assess- 
ment, and the project suggests that discussion of assessment is a particulariy 
helpful way of focussing students understanding of outcomes. 

The outcomes can provide the basis for the development of an overall assess- 
ment strategy, avoiding a cumbersome and piecemeal 'outcome by outcome* 
g^proach to assessment. Chapter 4 provides some examples of assessment de- 
velopment and discusses some of the questions and problems raised at this 
stage. 

At the end of the assessment stage it may be necessary to review the outcome 
descriptions, since the consideration of assessment is likely to have shed new 
light on them, and they may need some redrafting. 

Stage 3 - Modelling 

This stage relates the learning outcomes to a model of the * curriculum'. The 
stage is important because it shapes a potentially fragmented list of outcomes, 
of varying sizes and kinds, into a coherent model of the award. As many re- 
spondents to What Can Graduates Do? commented, the whole may be more 
than the sum of the parts, and a long and detailed list of outcome statements 
without structure may be unmanageable . 

The stage is not only important in the development of courses, but in creating a 
model of a decree for credit accumulation purposes. It helps to recognise 
diverse notions of coherence, and the range of value systems and purposes 
which exist within higher education, and among its client groups. 

The model can vary greatly in form and can relate outcomes of different kinds. 
The project identified several different models, which are described in Chapter 
6. 

The modelling process proviaes a way of determining the relative value of the 
various kinds of outcome of the programme. The views of employers can pro- 
vide a helpful input at this stage, in helping to define the *core' outcomes and 
the balance between the elements of the award. This is also a stage at which 
outcomes derived or taken from National Vocational Qualifications could be 
incorporated into a degree. 

Stage 4 - Credit Rating 

This stage relates the outcomes from the model to the CNAA CAT credit 
scheme. This raises the question of levels within the scheme, and how they are 
defined and understood. Chapter 5 provides some suggestions, but overall there 
is a need for greater clarity in this area, particularly in the distincton between 
learning at Level M and at other Levels. The relationship between CAT levels 
and the levels described by NCVQ also needs further investigation and clarifi- 
cation. 
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8.2 Measuring Quality 

8.2.1 Establishing criteria 

The evidence that a learning outcomes approach is useful is drawn from sev- 
eral quarters, from the responses to the consultative docunients and exercises, 
from the staff involved in the work and from the several institutions and or- 
ganisations which have pursued the development of learning outcomes in paral- 
lel with the project, or following on from it. 

The measurement of quality in higher education, has traditionally been difficult 
to separate from the processes of course delivery and the tacit internal values 
shared by academic staff, and it has been hard for those who are not members 
of the culture to gain access to, or understand the criteria by which quality is 
judged. Learning outcomes have provided a means of exploring and describing 
these criteria and of relating them clearly to assessment and to credits. This has 
helped to establish better criteria through which the quality of the teaching and 
learning processes in higher education could be measured. 

8.2.2 Course Documentation 

One of the more worrying fmdings of the project was that existing course do- 
cumentation prepared for the purposes of validation and review did not prove 
helpful to many of the groups in describing either learning outcomes, or assess- 
ment strategy. The descriptions of aims and objectives in many course docu- 
ments was very vague, and the documents did not, in many cases, relate the 
objectives to assessment, many merely stated the relative weightings for 
coursework and examinations. 

If course documents were to provide details of learning outcomes, this would 
of necessity provide information on what was being assessed, and this could in 
turn be related to the perfomance of students on the course. It would therefore 
be easy to see whether certain learning outcomes were consistently not 
achieved, or not achieved by some groups of students. This could be helpful in 
providing a basis for reviewing the teaching and learning methods employed by 
the course. Existing approaches to course documentation provide details of ob- 
jectives, but since these are not related directly to assessment, they cannot be 
related to student achievement. 

8.2.3 Student Voices 

The model has clear benefits for students. Learning outcomes provide a means 
of making clear to students what they need to learn and how that learning will 
be assessed, this not only helps to legitimate learning which takes place outside 
the HE institution, but it also empowers students by enabling them to focus 
clearly on what is to be learned and how the learning is to be demonstrated. 
This seems likely to foster and encourage *deep\ rather than 'surface* learning 
and could be used to help to improve the quality of student learning in higher 
education. Current and proposed increases in student numbers in higher ecuca- 
tion pose problems for the traditional teaching and learning strutures, which are 
"characterised by a tendency towards inexplicitness and a reliance upon tacit, 
traditional conventions'' (Wright 1989). The use of learning outcomes may 
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help students to develop more independent and active approaches to learning, 
and to be less reliant on tutor delivered teaching structures. 



6.2.3 Employer Voices 

The project provided a means of addressing the relationship between vocation- 
ally related occupational standards developed for NCVQ, and the academic 
standards of higher education. This will help to provide the means of integrat- 
ing and relating credits gained through the the NCVQ and the CAT credit 
schemes. 

The employer responses are quite clear that learning outcome descriptions can 
provide a useful veliicle for discussion with employers, but that it is the core 
outcomes which are of most interest to them. The work of the Enterprise in 
Higher Education Programme is exploring this territory helpfully. 
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Notes 



Note 1 ' Taxonomies, Learning Styles and 
Knowledge 

In the course of the project a number of approaches to the description and 
classification of learning outcomes, and the relationship of learning to perfor- 
mance, were examined. The following were all found useful in the definition of 
learning outcomes, although no single model addressed all the issues. 

The Bloom Taxonomy 

Bloom developed his taxonomy in the USA in the 1950s and it has been 
widely used ever since. It describes itself as a "classification of the student 
behaviours which represent the intended outcome of the educational process 
in the cognitive affective and psychomotor domains" . The cognitive taxo- 
nomy, which describes the objectives relating to knowledge, intellectual abil- 
ities and intellectual skills, was the most immediately relevant to the project. It 
describes a hierarchy of six categories: 

knowledge 

comprehension 

application 

analysis 

synthesis 

evaluation 

Each of which is divided into a set of sub-categories. These objectives (they 
arc in fact the action verb in an objective statement) are intended to be applied 
across a range of subjects and levels. 



23 The use of the word catcomc is interesting, but in tlie context of the remaining text it is 
clear that what is being described is what world normally be understood as an objective 
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The Carter Taxonomy 

Carter developed a taxonomy expressed m terms of the "personal qualities to 
be possessed by students, rather than the learning experiences by which they 
are to be developed*, (note that, like Bloom, Carter uses the language of learn- 
ing outcomes, but concentrates more on the ways on which tlie objectives are 
to be learned and the stmctures required for their development, than it does on 
the assessment of the objectives). It includes three types of objective: 

personal qualities, 
skiU, 

knowledge. 

Each of these is sub-divided into a series of sub-headings across what the 
author describes as, a "cognitive-affective axis". 

Kolb*8 Learning Styles Inventory 

David Kolb (1981) offers one of the most helpful and intelligible approaches to 
categorising knowledge in different disciplines. He used a Learning Style In- 
ventory (LSI) with a large group of managers. Essentially this measures 
preferred learning styles along two dimensions, concrete-abstract and active-re- 
flective. Kolb's work showed that the LSI scores were strongly correlated with 
the undergraduate majors of the people measured, and that these could be 
mapped into four types. 

Abstract/reflective 

(hard pure science, e.g. mathematics and natural science), 
Abstract-active 

(hard applied science e.g. engineering). 
Concrete-active 

(soft applied e.g. education and law) and 
Concrete-reflective 

(soft pure e.g. social science and humanities). 

Kolb's findings agreed well with those of other authors, notably Biglan 1973 
(who derived the hard-soft, pure-applied distinctions included above). Becher 
(1989) used these categories in a study of academic disciplines and he uses 
these observations to draw distinctions between the disciplines. For example, 
''Scientists learn about nature by seeking regularities and framing mathemati- 
cal models. There is little scope for patterning and reproducibility in the soft 
pure domain. To make the point another way, scientific knowledge is con- 
cerned primarily with universals, non-scientific knowledge tends to be focused 
on particulars**. 

This project has only touched the surface of these questions and the develop- 
ment of detailed and meaningful learning outcomes which capture these dif- 
ferences must be addressed in future work. 
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Eraut*8 Six Typc8 of Knowledge 

Michaei Eraut (1990) describes six types of knowledge derived from a study of 
management. However, he believes that the model has relevance in other fields. 
The six types arc: 

situational knowledge: 

describes the knowledge which people bring to the way they 
describe situations, it includes what they see, what they look 
for and how they interpret what they see. It also implies some 
elements of what might be called interpersonal skill in reading 
the behaviour of individuals in the situation. 

knowledge of people: 

includes not just existing knowledge of people, but also the 
ways in which a person gets to know people and makes 
judgments about them, including prejudices related to race and 
gender. 

knowledge of practice: 

includes knowing or being able to access factual infomiation 
in Older to take action, this further implies knowing several 
possible courses of action and being able to choose the most 
appropriate. 

conceptual knowledge: 

is concerned with infonnation handling and with how 
infonnation is recalled and related, this is related to what are 
described as cognitive frameworks or structures of kno^^'cdge 
(Wolf, in Wolf and Black 1990). These are also related to 
what Argyris and Schon (1974) described as theories in use 
and espoused theories. They argue that professional action is 
based on a series of implicit theories (theories in use), most of 
which have not been consciously worked out, and attempts to 
improve professional performance must depend on helping 
people to recognise their theories in use and bringing them 
under COTscious control. This is complicated by the use of 
espoused theories, which people use to describe th.j:ir actions, 
but are not necessarily the same as the theories wWch they use 
in practice. 

process knowledge: 

concerned with how to go about getting things done, includes 
logical sequencing skills like timetabling and budgeting as 
well as *thinking on your feet' and co-ordinating several 
different activities at once. 

control knowledge: 

is knowledge of one's self and one's thinking, it includes self 
awareness and sensitivity, and the gap between what one says 
and does. 
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Klemp and McLelland - Graduate Competences 

Klemp and McLelland developed an approach to describing competences 
through the systematic study of the differences between exceptional and aver- 
age performers. (Klemp. 1974, and Klemp and McQelland 1986). In a paper 
entitled ''Three Factors of Success'\ Klemp states " our most consistent, 
though unexpected, finding is that the amount of knowledge one acquires of a 
content area is generally unrelated to superior performance in an occx nation, 
and is often unrelated even to marginally acceptable performance''. 

Klemp identifies three factors of success in graduates: cognitive, inter-personal 
bnd intra-personal skills. These are similar to the types of knowledge described 
by Eraut. He suggests that most of the knowledge acquired in ^college' is lost 
very rapidly, but tiiat the three factors which are retained contribute to the 
success of the individual. The cognitive skills include: 

conceptualising skills, the ability to see thematic consistency 
in diverse information, and to organise and communicate this, 

the ability to understand many sides of a complex issue and to 
resolve informational conflicts, 

the ability to learn from experience, to translate observations 
from work experience into theories that can be used to 
generate behavioural alternatives, 

as well as a number of personal and interpersonal skills. 

He suggests that these 'skills' require more precise definition than is often ac- 
corded by academics. He uses communication skill as an example of a *com- 
pctence' which can be "so broad as to encompass all observable behaviour, and 
so narrow as to include only speaking and writing skills". Motivation (an intra 
personal skill) is also used as an example of a poorly defined skiU which can 
be taken to include everything a person does. He suggests that there are certain 
aspects of motivation which can be described with reference to "habitual 
thought patterns" for example, proactivity is associated with seeing oneself as a 
link in a 'cause and effect chain'. 

The Klemp work has been devr ' in British higher education through two 
pieces of work by UDACE wh • /: >red their relevance to admissions selec- 
tion for higher education. The r • * ' ed an American model in a British con- 
text (see Otter S. Student Potenn. m Britain). The second developed a British 
competence model (see Otter S. Admissions to Science and Engineering: an 
Admissions Tutors Handbook). 



Hi 



ERLC 



i-17 



101 



Notes and Appendices 



Note 2 - Aims and Objectives of Courses 

CNAA Principle 3 

1 Courses acceptable to CNAA must have aims and objectives which 
the curriculum structure, teaching methods and fonms of 
assessment are designed to fulfill.. 

2 The aims will include the development, to the level required for the 
award of a body of knowledge and skills appropriate to that field 
of study and reflecting academic developments in that field. 

3 ' The aims will also include the CNAA*s general educational aims; tiic 

development of students*: 

intellectual and imaginative powers; 
their understanding and judgment; 
their problem solving skills; 
their ability to communicate; 

their ability to see relationships within what they have learned 
and to perceive their field of study in a broader perspective. 

Each student's programme of study must stimulate an enquiring, analytical and 
creative approach, encouraging independent judgment and critical self aware- 
ness. 



Note 3 - Approaches to Assessment 

The problems which arose over assessment are discussed in Chapter 4. Some 
of the difficulties arose as a result of confusions about different kinds of as- 
sessment, and issues of grading, sampling and validity. This note is intended to 
clarify some of these points. 

Norm referencing 

Norm referenced approaches to assessment were cleariy the most familiar in 
higher education. Here a mark or grade indicates a candidate's attainment in 
relafion to other candidates in the cohort. The reference point against which a 
candidate is marked is how other candidates normally perform, there is no 
fixed external standard against which to assess each individual. 
V 

Criterion referencing 

In criterion referenced assessment the attainment of the candidate is measured 
against a set of pre-defined standards . This approach differ from norm refer- 
enced approaches in several important ways: 

• criterion referenced approaches have an externally defined standard, 
candidates are measured against the standard, not against one another; 
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• normally, in order to achieve credit a candidate must satisfy ALL the 
elements within the unit and must provide evidence which satisfies ALL 
the performance criteria; 

• criterion referenced approaches do not usually allow for grading. A 
candidate either does or does not satisfy the criteria. Exceptional 
performance is demonstrated either by the achievement of more criteria or 
criteria at a higher level. 

NVQs are a good example of criterion referenced qualifications. 
Pereonally referencing 

A personally referenced system measures the progress of the candidate in rela- 
tion to his or her starting point, there are no pass fail standanjs, or normative 
grades. Neither the absolute position of the learner, nor his/her position relative 
to others is important here, the focus is the progress of each individual along an 
axis like, for example, increasing autonomy. This approach allows learners at 
many different stages to learn collaboratively (there is no competition for first 
place). However, it is not possible to use such a system to award credit, which 
must, by definition, represent the achievement of some form of external stand- 
ard. 

"Sampling" 

In view of the complexity and variety of what higher education seeks to 
achieve, assessment is normally based on some form of sampling. Candidates 
are assessed on only a part of what they are believed to have achieved, and it is 
presumed tliat this sample accurately represents the whole. Tlius, typically a 
candidate completes a written paper where s/he has a choice (three questions 
from ten). The validity of such methods depends, of course, on the appropriate- 
ness of the sample, and this strategy is sometimes thought to imply that the 
candidate only needs to know part of the subject material taught on the course. 

The danger of 'sampling error' in this situation is compounded by tlie fact that 
degree examinations normally have a pass mark which may be set at 40% or 
50%, again perhaps implying that the candidate need only know some things 
about the subject being tested, and need not know them very well. These add to 
the problem described by many academic staff that students can both avoid 
some areas of the syllabus altogether and can get away with a 'relatively sket- 
chy* knowledge of the rest A frequent complaint of academic staff is the tend- 
ency of students to ' le rather then process what they learn. 

There are further problems in this type of assessment, these include: 

• the narrowing of student learning to focus on what will be assessed. 
Strategies adopted by students include question spotting and parrot 
learning of model answers, both approaches tending to encourage shallow 
rather than deep approaches to learning. 

• inconsistency in marking and grading. There is considerable evidence in 
the literature ^"^to show that the assessment of the same piece of work by 
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different people can result in a range of grades ranging in the extremes 
from First to fail. 

• the artificial nature of the assessment. Unseen written examinations are 
said by many to be "a good way of preparing young people and of 
sorting out those who will be able to cope'' (Jessup 1991), They do 
however provide an entirely artificial situation, few graduates ever 
subsequently have to write in this way, in general most are required to 
provide succinct reports (their lack of expertise in doing this has been 
reported by many employers and authors), rather than essay style papers. 
There are also few occasions in work when a piece of written work 
cannot be redrafted, perhaps in consultation with a colleague and certainly 
with reference to other material. The pressure which examinations place 
on many individuals contributes further distortion, 

VaUdity and rcUablUty? 

The prime requirement of any assessment process is that it measures what it is 
intended to measure. This issue of validity must take precedence over other 
technical issues, however, important. However, it would appear thai concern 
about reliability - the extent to which the process produces consistent results, 
sometimes absorbs disproprionate attention. 

Professional judgement 

Concern has been expressed about the danger that criterion referenced testing, 
which is inevitably associated with an outcome led approach, will lead to the 
measurement only of those things which are easy to test by simple mechanical 
process. Although this is a legitimate anxiety about the likely reality, there is 
no logical connection between the two. Any form of assessment ultimately 
rests on a professional judgement. The process of defining outcomes makes it 
easier to examine the underlying assumptions of those who assess, and explore 
how far they are achieving their puirposes, it does not need to imply a simplis- 
tic "testing" approach. 

Note 4 - Some notions of 'core' 

The project work clearly supported the notion of a set of *core' skills or out- 
comes which were generally expected of all graduates. There is no generally 
agreed definition of these qualities in higher education, although some notion 
of core clearly plays a part in selection at entry to HE as well as at graduation. 

The project identified a group of such qualities by extracting common elements 
from the work of the subject groups, but it also examined a number of other 
models. Some of these are outlined below. 

Government Core Skills 

The Government has specified a set of core skills to be included in A and A/S 
Levels and NVQs (NCVQ 1991), These are: 

problem solving: covering responses to problems encountered 
in a wide range of circumstances. The model recommended by 
NCVQ comprises three key features, whether the problem is 
already defined, or requires clarification, whether the approach 
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to ihc problem is provided or selected, and whether the 
solution is judged against criteria provided for or devised for 
the individual solving the problem. 

communication: includes interpreting information and 
presenting information and ideas. It involves reading writing, 
speaking and listening. 

personal skills: (also described as personal effectiveness and 
interpersonal skills) includes improving one's own 
performance and working with others. 

numeracy: focuses on the applications of number, it includes 
gathering and processing data, representing and tackling 
problems and interpreting and presenting data. 

information technology: differs fundamentally from the 
previous four, and is based on the use of IT to perform 
practical tasks. 

modern language competence: again differs from the first 
four, and may be best approached through the standards which 
will be developed by the languages lead body. 

KMI Core Skills 

HMI analysed a range of approaches to core, used in Youth Training, BTEC, 
CPVE and TVEI and identified the following list (HMI 1989): 

knowledge about IT 

knowledge about society and the environment 

communication 

numeracy 

using IT 

problem solving 

practical skills 

working cooperatively 

adaptability 

BTEC Common Skills 

BTEC have produced a list of common skills, to be included in all BTEC 
programmes (BTEC 1991). ITiese are: 

managing and developing self 
working with and relating to othei3 
communicating 

managing tasks and solving problems 
applying numeracy 
applying technology 
applying design and creativity 
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CBI Common Learning Outcomes 

The CBI defined a set of core elements which ihey wished to see in all traning 
and vocal -^nal education (CBI), These were: 

values and integrity 

effective communication 

applications of numeracy 

applications of technology 

understanding of work and the world 

personal and interpersonal skills 

'problem solving 

positive attitudes to change 

UDACE Key Abllites 

Building on the work of Klemp referred to in Note 1, UDACE carried out an 
empirical study of Science and Engineering students in Glasgow Polytechnic 
and Straihclyde University to produce a profile of key abilities (Otter 1991), 
This profile included: 

concern for standards 

time management 
pro- activity 
systematic thinking 
conceptual thinking 
information seeking 
flexibility 
self-confidence 
accurate self assessment 
leadership 
persuasion 
perception. 
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Note 5 - The Three CAT Systems 

There are three established mechanisms for credit accumulation and transfer in 
post-school education and training in England and Wales. All provide a frame- 
work of levels within which qualifications, or parts of qualifications can be 
placed and related to each other. The three systems are: 

National Vocational Qualifications: provide an outcome 
based framework for all vocational qualifications (supemsed 
by the National Council for Vocational Qualifications 

CNAA CATS: provides an award based framework for 
Bachelor's and Master's degrees (supervised at the time of 
writing by the Council for National Academic Awards, 
although the functions of CNAA are being redistributed 
following the Further and Higher Education Act 1992) 

National Open College Network: provides an outcome based 
framework for adult learning outside the scope of the formal 
examining and awarding bodies (NOCN is a voluntary 
association of individual OCNs, wliich exists to ensure 
consistent application of the national framework across 
individual OCNs) 

The three schemes are distinct in a number of ways, although attempts are 
being made to link them. The relationship between CNAA CATS and NVQs is 
being explored in relation to professional qualifications, while OCNs accredit 
the m; 'ority of Access courses, and work is in hand to link OCNs to NVQs. 

National Vocational Qualifications (NVQs) 

In the NCVQ framework the word credit is used only in relation to the accu- 
mulation and transfer of 'credit' gained for completion of u'^its. The structure 
uses tlie *unit' as its lowest common denominator: the smalles. part for which a 
separare certificate can be awarded. A unit is, however, not defined as a stand- 
ard siz/i, it must simply be a discrete aspect of competence, which makes sense 
as a free standing whole in the occupational area. The same unit may appear in 
more than one award, and the Awarding Bodies are required to recognise each 
other's Units towards relevant awards. Thus a candidate may achieve a unit of 
a qualification with City and Guilds, and then add this to units of a BTEC 
qualification to complete a BTEC award. 

A National Vocational Qualification (NVQ) which is the basis of an award, is a 
collection of units of competence, again not a random collection, but a coherent 
description of the competence standards for a particular occupation or role 
within an occupation. Neither an NVQ nor the units of which it is comprised is 
a course or a module. Courses may be constructed which help learners to reach 
these standards and through which the learners may be assessed, but NVQs are 
not courses and do not therefore refer to teaching or learning time, either no- 
tional or real. The intention is that many people will be able to demonstrate 
that they are competent to the specified standards thiough prior experience or 
training, without having to take a further course. 
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The new General NVQs (NCVQ 1991) represent an interesting departure from 
this position, GNVQs will be unit based but will be equivalent in size to A and 
A/S Levels, in order to provide an alternative, vocational, route into higher 
education. It appears that the General NVQ will not be made up of the same 
sort of occupational standards as the NVQ, although they will have a clear 
vocational purpose. This will be a basis for the Advanced Diploma, which may 
include A and A/S Levels as well as NVQs and General NVQs. This could be 
a very appropriate combination for a higher education candidate with clear vo- 
cational objectives. 

The CNAA CAT Scheme 

The CNAA CATS framework is built around the Honours degree, and each 
credit is essentially a fraction of what a student must do to achieve i. named 
award (degree, diploma, certificate etc, the Honours degree being 'worth' 360 
credits, 120 at each of three levels). The scheme also provides for a number of 
intennediate awards and for awards at post graduate Masters level (level M). 

What makes up a credit, or credits, is determined by individual HE institution, 
but the worth or value of the credit (V36O of the degree) is assumed to be the 
same in each. The scheme relies on the assumption that degrees in different 
subjeas and from different institutions are at the same level and standard A 
degree may then be made up of credits gained from several different institu- 
tions, or from credits gained in different subjects. However the learning indi- 
cated by the credits must have a clear sense of coherence and students cannot 
simply accumulate credits in a *pick and mix' fashion to achieve a named 
awaid. Although an individual may accumulate any number of 'general cre- 
dits', these have no value until they are accepted towards a specified degree, at 
which point they become 'specific credits'. 

National Open College Network 

The National Open College Network (NOCN) provides a national framework 
for the work of individual regional Open College Networks (OCN). Each OCN 
is a collaborative group of education and training agencies across a region, 
whose members submit their programmes (normally locally designed for par- 
ticular student groups) for credit rating within the nationally agreed framework. 
Programme submissions are based on descriptions of outcome, but the OCN, in 
recognising a programme also considers the relevance of the programme to its 
target group, its learning and assessment processes. While the programme's 
value is assessed on the basis of its outcomes, the number of credits awarded to 
learners on the programme is calculated on the basis of 'notional time' - the 
time required by the average learner, in normal circumstances to complete the 
programme - in imits of 30 hours. 

OCN's recognise programmes outside the scope of existing awarding bodies, 
up to degree entry level. They currently act as Accredited Validating Agencies 
for the majority of Access Courses, under the CNAA/CVCP Access Course 
Recognition process. Work is also in hand to enable learners to acquire Units 
of NVQs within OCN programmes. 

OCNs normally include in their membership agencies from public, private and 
voluntary sectors, including institutions of further and higher education, and 
they agree to accept for transfer purposes any credit awarded by a member 
institution (within any OCN recognised by the National Network). 
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Differences 

The most important difference between the three framewoiics is that they derive 
their authority and notions of value from different sources. CNAA CATS 
begins from the notion of the award (the Honours Degree), and is answerable 
to the quality assurance processes of higher education. NVQs, on the other 
hand are based on occupational competence, an employer view of what is re- 
quired for competent performance in the woiicplace, based on specified out- 
comes defined nationally. OCN credits are based on a 'Tx)ttom up" negotiation 
of equivalence between locally developed learning programmes, linked, where 
possible, into other credit and award systems. NVQs and OCNs both define 
outcomes, only OCNs make explicit use of notional time to measure credit 
value, although many higher education institutions do refer to time when allo- 
cating credit within the CNAA CATS scheme. 

These are fundamental differences and reflect the different purposes of the 
schemes. However the differences do not preclude a greater degree of integra- 
tion. It is not difficult to see how an NVQ unit, at an appropriate level and in 
an appropriate occupational area, could be included in a degree programme, 
perhaps as part of a sandwich course, and given a CAT credit rating derived 
from a judgment of the type of outcome and an overall balance of the student 
programme. The fact that the NVQ is defined in terms of outcomes and perfor- 
mance criteria makes this easier. The proposals in this report for distinguishing 
types of outcomes within a coherent degree programme would help in this. 

Degrees which are specifically a preparation for a particular area of employ- 
ment, or which lead to a professional qualification will be affected by the intro- 
duction of NVQ standards above Level 4. Current work in this area suggests 
that NVQs at this level will look rather different from the more familiar ones at 
Levels 1 and 2, and that they will address in some depth the knowledge and 
professional practice questions which engaged the attention of the subject 
groups in this project. It will also make sense for some degree courses which 
may have a vocational * flavour*, to include some relevant NVQ units as part of 
their courses. Here the differences between the NVQ unit and the CAT credit 
will be an advantage, it will be possible to incorporate the NVQ standards into 
a degree course structure as general CAT credits, and to provide a CAT credit 
rating based either on the notional hours of learning, or on the relative con- 
tribution which the NVQ unit makes to the whole award. This does not imply 
that a degree will become an NVQ, and the two are cleariy different and serve 
different purposes, but there are and will be areas where the two can overlap 
and where this will be a benefit to the student. 

A national framework 

The development of a national framework for credit accumulation and transfer 
will call for the harmonisation, in some way, of the work of CNAA CATS, 
National Vocational Qualifications (NVQs) and Open College Networks 
(OCNs). This would provide a system of credits extending across the whole of 
post school education, from adult basic education to Masters degree, and offer- 
ing the means of progression for adults and young people at any stage in tiieir 
career. The integration of these different systems will be greatiy facilitated by 
the clear description of what students gain credit for. 
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Appendix 1 



Project Strategy and 
Methodology 

The work of the project centred on three groups of people, academic staff, 
students and graduate employers, this section describes and briefly evaluates 
how the work was carried out and the contribution it makes to the overall 
findings. The chart below outlines the stages of the work with each group. 





Employers 


Academic Staff 


Students 


1 




Institutional 
Consultation 
(November 1989) 




2 




Selection of Institutions 
(December 1989) 




3 


Employer 
Seminar 1. 

Employer 
Visits 
(April 1990) 


Subject Group Work 
(Jan-Dec 1990) 




4 




Consultative Seminar (Sep- 
tember 1990) 


Student 
Workshops 
(Nov-Mar 91) 


5 




Assessment Workshop 
(Sept 1990) 


Assessment Pilots 
(Sep-Jun 1991) 


6 


Employer 
Seminars 
(Feb-May 
1991) 


Consultative Document 
What Can Graduates Do? 
to all HEIs 
(Jan 1991) 




7 


Employer Con- 
sultative Docu- 
ment 
Issued 
(May 1991) 


Review Workshop 
(May 1991) 




8 




Draft Report to Steering 
Committee (Oct 1991) 
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Stage 1: Initial Consultation 

The initial consultative seminar was held All HE institutions in England and 
Wales weic invited, and 23 attended. The seminar was intended to do two 
things: 

• firstly to inform as many people as possible of the project and its aims 
and objectives, 

• secondly to assist in deciding which subjects would be the most suitable 
for detailed study in the project. 

The exercise provided information about the subject areas which were felt to be 
the most readily investigated, and confirmed that there was considerable inter- 
est in developing a learning outcomes approach to courses in higher education. 
The five subject groups chosen by the Project Steering Group were: Design, 
Engineering, English, Environmental Science and Social Science. . 

Stage 2: Selection of Institutions 

Letters inviting institutions to bid to participate in the project were sent to all 
univcrsitites. Polytechnics and Colleges of HE in England and Wales. 53 in- 
stitutions responded to the invitation and eleven were selected to take part. Two 
of the latter (Bradford University and Leicester Polytechnic) subsequently with- 
drew. 

Each institution provided academic staff who were seconded on a half time 
basis for one year. TTiese people made up the five subject working groups 

The groups were deliberately composed of lecturers in higher education drawn 
from a range of departments in both Universities and Polytechnics. They repre- 
sented a fairly typical cross section of staff, with experience in teaching, as- 
sessment, course design, admission and recmitment of students, work witli 
adult students and research in their subjects. None of the members of the group 
had a background in educational theory and almost none of them had heard of 
learning outcomes before Jie project began. 

There were two main reasons for creating the groups in this way, 

• the outcomes of the work of such a group would give a good indication 
of how the development of learning outcomes might be undertaken on a 
larger scale by other academic staff in the future, 

• the findings of such a group were likely to be taken seriously as 
representing the views of mainstream academic staff rather than the views 
of a group of experts in educational development or the Employment 
Department. 



25 Apjpendix 2 lists the staff participating in the groups. 
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Stage 3: Subject Group Work 
Objectives 

The groups were given three specific objectives for the twelve months work, 

• to develop a set of learning outcomes for their subjea area, with drafts by 
September 1990 and final versions for circulation by November 1990, 

• to develop a way of accrediting the learning outcomes, 

• to pilot appropriate methods for the assessment of the learning outcomes. 

One objective was to investigate a range of approaches to describing md writ- 
ing learning outcomes. This was one reason for establishing different subject 
groups and for selecting a range of subjects which included cleaily vocational 
subjects like Engineering and * academic' subjects like EngHsh. 

Encouraging Diversity 

It was anticipated Uiat the different structures and traditions of the five subjects 
might lead to different ways of describing and structuring learning outcomes. 
The subject groups were therefore actively encouraged to develop individual 
approaches to describing their learning outcomes and to concentrate on develo- 
ping structures which seemed most appropriate for Uiem. 

The groups were encouraged not to begin by developing learning outcomes for 
different years or stages of courses, but to describe tiie outcomes tiiey sought at 
the final degree stage, seeking to define what tiiey expected tiieir graduates to 
know and be able lo do. 

There were two reasons for taking this approach, 

• tiiinking of the outcomes in levels was likely to confine tiie groups to 
considering tiie structures of tiieir existing courses. This seemed likely to 
lead to what one member described as " a cynical rewriting of course 
objectives in Uie passive tense", 

• tiie courses represented were very different in structure and philosophy, 
for example material covered in tiie first year of some courses was 
included in tiie final year of otiiers, some courses tackled broad principles 
at an early stage while others sought to develop tiiese over tiiree years. 

All tiie staff were provided witii a detailed briefing on tiie project and back- 
ground papers, and a wide range of articles, papers, books and otiier publica- 
tions were circulated to tiie group members. A series of workshops was also 
arranged to provide experience of tiie Employment Department's fimctional 
analysis approach to defining outcomes, on employer views of graduate learn- 
ing outcomes and on assessment and credit structures. The groups worked in- 
dependently for most of tiie time, arranging small group meetings and dividing 
specific tasks between members. Notes of all meetings w\.ie circulated botii as 
reminders of actions agreed and as a means of establishing a consensus. All 
five groups met together on a number of occasions for workshops and to re- 
view developments across all tiie groups. 
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Stage 4: Consultative Seminar and Workshops 

A major seminar workshop involving 93 academic staff was held in September 
1990 in Sheffield Representatives of all higher education institutions were in- 
vited to attend and draft papers were sent to all those who replied. The work- 
shop was divided into subject groups and delegates were able to attend two 
workshop sessions during the day. 

The seminar and the workshops provided an opportunity for critical peer re- 
view of the drafts. There was considerable, and critical comment on the ma- 
terial provided, and several of the project subject groups rewrote their drafts 
prior to publication of What Can Graduates Dol The Engineering group was 
particularly criticised for the very faaual and knowledge based nature of the 
outcomes described. Environmental Science, on the other hand was criticised 
for the general nature of the outcomes in the draft, which were thought to be 
applicable to any graduate and not to provide either sufficient real substance or 
a flavour of what environmental science graduates were like. 

Other workshops were held throughout the project, and the ideas were debated 
by UDACE and institutional staff at a range of conferences and meetings to 
which they were invited by other agencies. 



Students 

Consultation with students was carried out through a series of workshops held 
with groups of first and final year students in all the participating institutions. 
A workshop approach was chosen as a practical way of seeking the views of 
students, since individual interviews or questionnaires were both thought un- 
likely to yield sufficient information. Woricshops offered a way of reaching a 
large group of students and of promoting some discussion between them of 
learning outcomes. 

44 workshops were held between November 1990 and March 199 L The aver- 
age number of students in each workshop was 16. A standard workshop proto- 
col was developed in order to increase reproducibility and make it easier to 
analyse the responses obtained. The workshop had the following structure: 

• students were given a probe questionnaire, which they were asked to 
complete individually. This comprised a number of statements with which 
students vvere asked to agree or disagree, these included statements 
relating to academic work (be able to think for myself), and practical and 
technical skills, as well as statements relating to work and careers, (I have 
career options I did not have before), and to personal and social aspects 
(meet interesting people). 

• they were then asked to woric in small groups and to agree on ten learning 
outcomes for their course, for first year students these were outcomes 
they expected to achieve and for final years they v/ere learning outcomes 
that students felt they had achieved. 

• these were then discussed by the whole group and areas of ambiguity and 
overiap examined. 

• data from questionnaires and from small and large group discussions was 
colleaed and analysed. 
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Students welcomed the opportunity to discuss their views of learning outcomes 
and to express what they were seeking from their courses. Some of the aca- 
demic staff working on the project expressed surprise that their students had 
such a wide range of views on their learning and were surprised at the depth 
and detail of the responses. Several staff reported the findings of the workshop 
as being valuable in terms of curriculum design and course development. 

The outcomes described by students were much broader than those described 
by the staff. Almost half the outcomes which students anticipated or felt they 
had achieved were in the personal and social area, whereas only one quarter of 
the outcomes described by staff were in this area. Details of the student out- 
comes are presented in Appendix 6. 

Stage 5: Assessment 

Each 'institution developed and piloted a method of assessing one or more of 
the identified learning outcomes with a group of students. This woric required 
staff to produce materials and often worksheets for students. Some of the exer- 
cises involved other academic staff in the departments. All the pilot exercises 
were evaluated by staff and students. 

Stage 6: What Can Graduates Do? 

Revised drafts of the learning outcomes developed by each group were publish- 
ed in a document entitled Wfm Can Graduates Do? which was sent for com- 
ment to all higher education institutions in the UK. Separate copies went to: 

the Vice Chancellor, Director or Principal, 

the Dean or Head of Department in each of the five subjects 
addressed by the project. 

A questionnaire outlining areas of response was included and replies were 
sought within two months. The 74 responses received included: 

responses following discussion in Departments, Faculty and 
Academic Boards and internal consultation of various kinds, 

responses from individuals, some commenting on the project 
as a whole and some on the learning outcomes developed in a 
single subject area. 

Consultation with Graduate li^ployers 

Consultation with graduate employers was achieved through the circulation of 
a revised version of What Can Graduates Do? Since the original text was 
written for academic staff and was generally felt to be too detailed to send to 
employers, it was revised and edited with the help of a small group of em- 
ployer representatives who had offered their support for the project. The em- 
ployers recommended that the document be widely circulated to the major 
graduate employers and to some smaller employers. They suggested that the 
most useful responses would be obtained from people who were line managers 
of young graduates and whose business success depended on recruiting good 
graduates. They felt that responses from personnel specialists were likely to be 
less informative. Copies of the document were therefore circulated with a 
covering letter inviting comment from line managers. 
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The revised version of the document was in two parts, the first included a table 
of 13 general learning outcomes based on the common elements of the out- 
come lists developed by the subject groups. The general outcomes were: 

access and select information 



synthesise and interpret information 



demonstrate commitment 



demonstrate seff discipline 



manage personal stress 



communicate clearly and accurately 



communicate effectively orally 



work cooperatively 

I 

1 work alone 



accept criticism 



understand own strengths and weaknesses 



act ethically 



possess basic computer skills 

The second part contained an edited version of the subject specific learning 
outcomes with spaces for comments and additions. It was circulated through 
the Association of Graduate Recruiters and to a seleaed group of Design em- 
ployers, Local Authority Social Service Departments and to some Personnel 
specialists through the Institute of Personnel Management. 

In all 800 copies of the document were circulated and 138 replies received. 

Appendix 5 lists the employers who responded. 

Stage 7: Final Woricshop and Report 

The project ended with a residential workshop for institutional staff, to review 
the responses to What Can Graduates Dol and discuss the general conclusions 
of the project. 

Sta^je 8: Report Preparation 

Following the workshop the Project Director prepared a full draft report for 
discussion with the project steering group collectively and individually. In the 
light of that consultation the present report was produced for publication by the 
Project Director, in consultation with the Head of Unit. 
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Subject Group Members 



Design group 

Liverpool Polytechnic 

Professor Roger Breakwell Dean of Faculty of Art and Design. 

John Barraclough Principal Lecturer, Course Leader, 

Graphic Design. 



Sheffield Polytechnic 

Professor David Vaughan 
Ian Rodger 
Peter Slater 
Sue Drew 



Head of Department. 

Senior Lecturer, Course leader 

Senior Lecturer Course Leader 

Careers/personal sldUs and qualities project 



Engineering 

City University 
Roy Winterbum 

Nottingham Polytechnic 
Dr Bob Munton 

Polytechnic of Wales 
Colin Rees 



Continuing Education. 



Mechanical Engineering 



Computing 



English 

Lancaster University 
John Mowat 

Newcastle Polytechnic 
Alan Ingram 

Polytechnic of Wales 
Rob Middlehurst 
John Beynon 



Senior Lecturer, Admissions Tutor 



Senior Lecturer, Course Leader 



Senior Lecturer, Communication Studies 
Senior lecturer. Communication Studies 



Subject Group Members 



Envtronmental 
Science 

Lancaster University 

Peter Larson Senior Lecturer, Admissions tutor 

Liverpool Polytechnic 

Dr Terry Jeeves Liverpool Integrated Credit Scheme 

University of East Anglia 

Dr John Barkham Senior Lecturer 

Frances Nicholas Research Assistant 



Social Science 

Newcastle Polytechnic 
Dr Roy Boyne 

Sheffield Polytechnic 
Dr Peter Ashworth 
Judy Saxton 



Head of Department 

Principal Lecturer 
Research Assistant 



Teeside Polytechnic 

Barry Sudworth Senior Lecturer 
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Consultative Seminar: 
Institutions Represented 



Sheffield aty Polytechnic 


Liverpool Polytechnic 


Open University 


Polytechnic of Wales 


Newcastle Polytechnic 


Council for National Academic Awards 


Middlesex Polytechnic 


University of Warwick 


Polytechnic of East London 


Bristol Polytechnic 


Cardiff lastitute of HE 


University College of North Wales 


Nottingham Polytechnic 


Manchester Polytechnic 


University of Surrey 


University of Bradford 


Training Agency 


Huddersneld Polytechmc 


Leicester Polytechnic 


Umversity of Keelc 


Scottish Wider Access Programme 


University of Hull 


University of London 


Bmnel University 


Staffordshire Polytechnic 


Polytechnic of North London 


Luton College of HE 


B.rmingham Polytechnic 


Wolverhampton Polytechnic 


Loughborough University 


Liverpool lastitute of HE 


Coventry Polytechnic 


Teeside Polytechnic 


Bolton Institute of HE 


Leeds Polytechnic 


University of Kent 


Lancaster University 


Oxford Polytechnic 


University of Durham 


Glasgow College 


Nene College 


University of Ulster 


Chester College of HE 


Anglia HE College 


City University 
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"What Can Graduates 
Do?' Institutions and 
Individuals Responding 



CNAA 

Robert Gordons Insitute of Tech 
King Alfred's College, Winchester 
Southampton Institute of HE 
Roehampton Institute 
Birmingham Polytechnic 
Bristol Polytechnic 
Humberside Polytechnic 
Polytechnic of Central London 
Manchester Polytechnic 
Newcastle Polytechnic 
Oxford Polytechnic 
Portsmouth Polytechnic 
Staffordshire Polytechnic 
Teeside Polytechnic 
Polytechnic of Wales 
The Open Univ. East Anglian Region 
The Open University, Scotland 
University of Belfast, Queen's 
University of Bristol 
University of Durham 
University of Edinburgh 
University of Lancaster 



Grays School of Art 

Falmouth School of Art and Design 

Liverpool Institute of HE 

Paisley College 

St Mary's College 

Bournemouth Polytechnic 

Glasgow Polytechnic 

Kingston Polytechnic 

Polytechnic of East London 

Middlesex Polytechnic 

Nottingham Polytechnic 

Plymouth Polytechnic South West 

Sheffield Qty Polytechnic 

Sunderland Polytechnic 

Thames Polytechnic 

Wolverhampton Polytechnic 

The Open University, Walton Hall 

University College of Ripon & York St John 

University of Bradford 

University of Dundee 

University of East Anglia 

University of Hull 

University of Leeds 
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University of Liverpool 

University of Leicester 

University of Newcastle upon Tyne 

University of Oxford, 
School of Geography 

University of Salford 

University of Stirling 

University of Wales, Bangor 

University of West London, Brunei 

Professor Lewis Elton 

Jane Hanx)p, DES 



University of Loughborough 
University of lx)ndctti, Wye College 
University of Nottingham 

University of Reading 
University of Southampton 
University of Ulster 
University of Wales, Cardiff 
University of York 
Rev. R Tuck 
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Employer Responses 



Alfred McAlpine Construction Ltd 

Bank of Scotland 

British- American Tobacco Co Ltd 

CMB Packaging 

Cameron Markby Hewitt 

Colas Holdings Ltd 

Creed Lane Graphic Communication 

Cumbria Constabulary 

Daiwa Europe Ltd 

Digitial Equipment Co Ltd 

Durham Constabulary 

Financial Solutions Ltd 

Ford Motor Co. Ltd 

Hadcn Young Ltd 

Herbert Smith 

Hitachi Europe 

Hymatic Engineering Co. Lid 

IDV UK Ltd 

Legal and General 

MOD, Defence Engineering Services 
& Defence Science Group 

Noble Lowndes and Partners Ltd 
Massey Ferguson Tractors Ltd 
PSA Services 
Pentagon Training Ltd 
Pilkington PLC 
RTZ Ltd 

" i.r,' 



Balfour BeaUy Ltd 
Barclays Bank PLC 
British Gas PLC 
Cadbury Schweppes PLC 
Cleveland Constabulary 
Courtaulds PLC 
Crosfield Electronics 
Dairy Crest Ltd 

Devon and Cornwall Constabulary 
Dixon Wilson 
Durham County Council 
Fisons PLC 

Guardian Royal Exchange 
Hampshire Constabulary 
Herts Constabulary 
Hoover 
ICI 

Institute of Chartered Accountants 
Lucas Industries PLC 
NNC Ltd 

Nottinghamshire Constabulary 

Newey and Gyre 

Peat Marwick McLintock 

Peugeot Talfcot Motor Co Ltd 

Prison Service 

Rank Xerox (UK) Ltd 
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Reeves & Neylan Oiartered Accountants 

Rolls-Royce 

Royal Ordnance PLC 

Seismograph Service 

South Wales Constabulary 

Steetley PLC 

T C Scheme 

Thames Valley Constabulary 
Unilever 

Warwicks Constabulary 
WH Smith Ltd 



Robson Rhodes 

Royal Bank of Scotland PIC 

Schlumberger 

Smith Kline Beecham 

Standard Life 

Suffolk Constabulary 

Tarmac Construction Ltd 

Thorn EMI Electronics 

VSO 

West Midlands Constabulary 
Yorkshire Water PLC 



There were also 36 responses from small employers in Design, gathered 
through one of the participating institutions. 
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The Student Outcomes 



The following learning outcomes were identified by student groups at the stu- 
dent workshops. The number of occasions on which woiicshops groups men- 
tioned each is shown, together with its orientatior (Personal/Social (P), 
Cunicular (C) and Vocational (V)). and examples of statements recorded . 



Outcome Type 

Enhanced career options 

Increased independence 

Increased self esteem 

Acquire skills/techniques 
Become more knowledgable 

Critical thinking 

Increased job satisfaction 
Acquire IT skills 
Increased understanding 



No. Orientation 
refs. 



Examples 



13^ 



60 V Will open up more career doors 
I will have career options that I 
didn't have before 

55 P more self reliant and responsible 
more independence from friends 
and family 

49 P become a stronger person. 

satisfaction from handling things 
better. 

46 C gain practical skills, not just theory, 
a broader base of skills. 

40 C wider knowledge of art history. 

gain more knowledge and broaden 
horizons 

39 C more critical and questioning 

things, develop a critical approach, 
more developed constructive 
critical skills 

34 V able to progress in my chosen 
career, ability to choose a job I 
enjoy. 

34 C able to use a computer, increased 

understanding of basic statistics and 
comupter packages 

34 C a greater understandiong of history. 

understand English literature more 
fully, some sense of my place in 
the universe. 
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Meet a wide range of people 34 

Improved written communications 3 1 

Improved oral COTimunications 30 

Make good friends 28 

Leam time management 28 

Become self critical 27 

Increased tolerance 26 

Acquire analytical kills 25 

Work with others 24 

Think for oneself 24 

Positive view of course 24 

Ability to make money 22 

Increased self -discipline 22 

Improved life skills 22 

Changed attitudes 21 

Flexibility in new situations 20 

Get a good degree 18 

Produce good work 16 

Negative view of course 13 

140 



P meet interesting pec^le. come 
across a wider social mix. get to 
know people from different 
backgrounds. 

C leam to write better, able to write 
clearly and fluently. 

P better verbal skills, better listener 

P good friends and contacts, meet 
people and make friends. 

P better organisation of time, time 
management. 

P know your own limitations, know 
and accept my limitations 

P get on with people from different 
backgrounds, increased 
understanding of people and 
individual circumstances 

C interpretation - not only of data but 
of situations, analyse information 

P capable of working for and with 
others, better work in groups. 

P better able to think for myself, 
greater chance of individual 
expression. 

C chance to study and leam about 
areas that are interesting. 

V make money, increased salary and 
better standard of living. 

P better prepared to stick at 

something, knowing when to work 
and when to play. 

P better organisation of finances. I 
can cook better, better organised for 
life after Polytechnic. 

P increased environmental awareness, 
will be more concerned about 
issues. 

P coping with areas of no previous 
experience, able to cope with the 
unknown. 

CA^ finish with a good degree, a first! 

C do oneself justice, produce good 
work 

C less idea of what 1 want to do with 
my life than when I started. 
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Improved social life 13 

Acquire problem solving skills 13 

Acquire management skills 12 

Go to parties 1 1 

Acquire industrial experience 1 1 

Increased altruism 1 1 

Acquire library skills 10 

Join in extracurricular activities 10 

Acquire data collection skills 10 

Increased confidence with others 10 

Increased motivation 10 

Get a degree 9 

Improved social skills 7 

Delay decisions about career 6 

Take more risks 5 

Increased vocabulary 4 

Opportunity to travel 4 



P meeting people and socialising, 
social position is much belter now. 

C more logical approach to problems, 
solve problems. 

CA^ capable of leading a team or 
workforce on projects, organise 
other people and delegate. 

P have a good time, three years of 
parties, increased tolerance to 
alcohol. 

V interesting year out in industry, get 
hands on experience. 

P to help people, iielping people to 
help themselves. 

C able to use libraries more 

effectively, belter able to extract 
information from books. 

P involvement in 

clubs/societies/politics, go 
mountainering. 

C design and carry out research, 
obtain data. 

P expect to be more socially 

confident gain more confidence at 
public speaking. 

P able to use own initiative, improved 
committment 

C a piece of paper, a recognised 
qualification. 

P able to mix better socially. 

V time to work out my future career, 
avoid going out to work. 

P able to take more risks, more 
adventurous. 

C increased vocabulary. 

C broaden horizons geographically, 
living in a different part of the 
country. 
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report. 
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What was UDACE? 

UDACE was a national development unit for the education of adults in England and Wales. It was 
created as a unit of the National Institute for Adult Continuing Education in 1984 by the Secretary of 
State for Education and Science. The Unit's role was to examine possible areas of development, 
recommend strategies for development and sponsor projects to support development. 

In AprU 1992, foUowing a decision of the Secretary of State, the Unit was merged with the Further 
Education Unit, which inhereits its work and agendas. During the Ufe of this project tiie Unit's 
Governing Body was a Steering Committee annprising the following members. 

Steering Committee Membership 

Chairman Donald Grattan 

Department of Education and Science 
Margaret d'Armenia John Steel HMI 

Welsh Office Education Department 
Owen Jones HMI 

Department of Employment 

Ros Seyd 

Association of County Councils 
John TumbuU 

Association of Metropolitan Authorities 

Alan Culley 

Welsh Joint Education Committee 
Gary Morgan 

National Institute of Adult Continuing Education 
Eddie Burch Hywel Francis Baibara Marsh Lydia Merrill HaroonSaad Jenny Scribbins 

Co-opted 

Ray Cowell Lucia Jones Peter Shaw Jenny Shackleton 

Ex-officio 
Eileen Aird Jonathan Brown 



Head of Unit Stephen McNair 
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